Transmitting apparatus, transmitting method, and receiving apparatus

ABSTRACT

Transmission of relating information of image data such as disparity information or the like is enabled to be performed favorably. 
     A data packet made up of a header portion and content portion is generated. Related information of image data, for example disparity information or the like, is inserted into the content portion of the data packet herein. Identification information to identify the type of related information is inserted into the header portion of the data packet herein. The data packet herein is correlated to image data, and transmitted to an external device. The related information of the image data such as disparity information or the like can be efficiently transmitted to an external device.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is a national phase entry under 35 U.S.C. §371of International Application No. PCT/JP2013/053852 filed Feb. 18, 2013,published on Sep. 6, 2013 as WO 2013/129158 A1, which claims priorityfrom Japanese Patent Application No. JP 2012-045912 filed in theJapanese Patent Office on Mar. 1, 2012.

TECHNICAL FIELD

The present invention relates to a transmitting apparatus, transmittingmethod, and receiving apparatus, and particularly relates to atransmitting apparatus and the like that can favorably performtransmission of related information that relates to image data such asdisparity information or the like.

BACKGROUND ART

For example, a transmitting method that uses television broadcast wavesof stereoscopic image data is proposed in PTL 1. In this case, left eyeimage data and right eye image data that make up the stereoscopic imageare transmitted, and stereoscopic image display using binoculardisparity is performed at the television receiving device.

FIG. 64 indicates, in a stereoscopic image display that uses binoculardisparity, the relation between the display position of the left andright images of the object (physical item) on a screen and the playingposition of the stereoscopic image thereof. For example, regarding anobject A displayed so that a left image La is shifted to the right sideand a right image Ra is shifted to the left side on the screen asillustrated in the diagram, the left and right lines of sight intersectnearer than the screen face, so the playing position of the stereoscopicimage thereof is nearer than the screen face.

Also, for example, regarding an object B displayed so that a left imageLb and right image Rb are displayed on the screen at the same positionas illustrated in the diagram, the left and right lines of sightintersect nearer on the screen face, so the playing position of thestereoscopic image thereof is on the screen face. Further, for example,regarding an object C displayed so that a left image Lc is shifted tothe left side and a right image Rc is shifted to the right side on thescreen as illustrated in the diagram, the left and right lines of sightintersect on the far side of the screen face, whereby the playingposition of the stereoscopic image thereof is on the far side of thescreen face.

CITATION LIST Patent Literature

-   PTL 1: Japanese Unexamined Patent Application Publication No.    2005-6114

SUMMARY OF INVENTION Technical Problem

As described above, with a stereoscopic image display, a viewer usesbinocular disparity to recognize perspective of the stereoscopic image.Now, the disparity angle corresponding to the nearest object playingposition (disparity angle in the intersecting direction) and thedisparity angle corresponding to the farthest object playing position(disparity angle in the same-side direction) needs to be within apredetermined range so as not to harm the health of the viewer. That isto say, the disparity angle is checked at the receiving device, and inthe case that the disparity angle herein is not contained within thepredetermined range, it is expected that the left eye image and righteye image will be reconfigured so as to fit within the predeterminedrange.

Also, regarding graphics of an OSD (On-Screen Display), which isdisplayed overlaid onto an image on a receiving device (set-top box,television receiving device, or the like), or of an application or thelike, it is expected that not only in two-dimensional space, but also asthree-dimensional perspective, rendering is performed coupled with thestereoscopic image display. In the case that graphics are overlaid ontoan image to display on a receiving device, it is expected that disparityadjustment will be performed according to the perspective for eachobject within the image, and that consistency of perspective will bemaintained.

The object of the present technology is to enable favorably performingtransmission of related information of image data such as disparityinformation or the like.

Solution to Problem

A concept of the present technology is in a transmitting apparatus thatincludes

-   -   a data packet generating unit to generate a data packet made up        of a header portion and a content portion; and    -   a transmitting unit to correlate the data packet to image data        and transmit to an external device,    -   wherein the data packet generating unit inserts related        information of the image data into the content portion, and        inserts identification information to identify the type of the        related information into the header portion.

According to the present technology, a data packet consisting of aheader portion and content portion is generated by a data packetgenerating unit. Also, the data packet herein is correlated to imagedata and transmitted to an external device by the transmitting unit.With the data packet generating unit, related information of the imagedata is inserted into the content portion, and identificationinformation to identify the type of related information is inserted intothe header portion.

For example, with the data packet generating unit, the size of thecontent portion is determined according to the data amount of therelated information inserted in the content portion, and the sizeinformation indicating the determined size herein may be inserted in theheader portion. For example, with the packet generating unit, the datapacket may be generated for each of a predetermined number of picturesof the image data. Now, in the case that the predetermined number is 1,data packets are generated corresponding to all of the pictures.

For example, the image data is left eye image data and right eye imagedata that configures the stereoscopic image, and the related informationis disparity information of the other as to one of the left eye imageand right eye image, and is set as representative disparity informationfor each predetermined region of the picture display screen. Forexample, in this case, a first disparity information corresponding tothe object playing position in the nearest of the predetermined regionis included in the representative disparity information for eachpredetermined region. Also, for example, the first disparity informationcorresponding to the object playing position in the nearest of thepredetermined region and a second disparity information corresponding tothe object playing position in the farthest of the predetermined regionare included in the representative disparity information for eachpredetermined region.

Thus, according to the present technology, related information of theimage data, for example, disparity information or the like, is insertedin the content portion of the data packet, while the identificationinformation to identify the type of related information is inserted inthe header portion of the data packet herein, and transmitted to anexternal device. Therefore, the related information of the image datasuch as disparity information or the like can be efficiently transmittedto an external device.

Note that according to the present technology, for example, in the datapacket generating unit, the representative disparity information may beinserted in the content portion as absolute value data. In this case,positive and negative coding bits are unnecessary, and the dynamic rangeof the disparity information can be expanded by just that much.

For example, the transmitting unit may be arranged so that the datapacket is inserted into a blanking period of the image data andtransmitted to an external device. Also, for example, the transmittingunit may be arranged so that transmission data is generated inincrements of video field segments that include horizontal blankingperiods and vertical blanking periods segmented by the verticalsynchronizing signal, and active video spaces having primary pictureregions and auxiliary picture regions, and transmitted to an externaldevice, where image data is distributed to the primary picture regionsand data packets are distributed to the auxiliary picture regions.

Also, a concept of the present technology is in a transmitting apparatusthat includes

-   -   an image data obtaining unit to obtain left eye image data and        right eye image data that configures a stereoscopic image;    -   a disparity information obtaining unit to obtain representative        disparity information which is the other disparity information        as to one of the left eye image and right eye image, for each        predetermined picture of the image data, and which is in each        partition region corresponding to a partition pattern of a        picture display screen;    -   a disparity information inserting unit to insert the        representative disparity information for each partition region        into the video stream obtained by the image data having been        encoded; and    -   an image data transmitting unit to transmit a container of a        predetermined format that includes a video stream in which the        disparity information has been inserted.

According to the present technology, left eye image data and right eyeimage data configuring the stereoscopic image is obtained by an imagedata obtaining unit. Representative disparity information that is theother disparity information as to one of the left eye image and theright eye image, and that is in each partition region corresponding tothe partition pattern of the picture display screen, is obtained by thedisparity information obtaining unit for each predetermined picture ofimage data. Also, representative disparity information in each partitionregion is inserted into the video stream where image data is encoded andobtained, by the disparity information inserting unit.

For example, a pattern selecting unit that selects a predeterminedpartition pattern from multiple partition patterns may be furtherprovided, where the disparity information obtaining unit may obtainrepresentative disparity information in each partition regioncorresponding to the predetermined partition pattern selected for thepicture display screen. In this case, a user can have a situation whererepresentative disparity information of each partition region from adesired partition pattern is obtained by the selection of the partitionpattern.

Also, for example, the first disparity information corresponding to thenearest object playing position in the partition region may be includedin the representative disparity information in each partition region.Also, for example, the first disparity information corresponding to thenearest object playing position in the partition region and the seconddisparity information corresponding to the farthest object playingposition in the partition region may be included in the representativedisparity information in each partition region.

Also, for example, the disparity information inserting unit may bearranged so that the representative disparity information is inserted inthe video stream as absolute value data. In this case, positive andnegative coding bits are unnecessary, and the dynamic range of thedisparity information can be expanded by an amount equivalent thereto.

Also, yet another concept of the present technology is in a receivingapparatus that includes

-   -   an image data receiving unit to receive a container of a        predetermined format that includes a video stream,    -   wherein the video stream is obtained by left eye image data and        right eye image data that configure a stereoscopic image having        been encoded; and    -   wherein representative disparity information which is the other        disparity information as to one of the left eye image and right        eye image, at each partition region corresponding to a partition        pattern of a picture display screen, is inserted into the video        stream for each picture of the image data;    -   the receiving apparatus further including    -   an information obtaining unit that obtains the left eye image        data and right eye image data from the video stream included in        the container, while obtaining representative information for        each partition region of each picture of the image data;    -   an information smoothing unit that performs smoothing processing        in the temporal axis direction as to the representative        disparity information for each partition region of each of the        picture;    -   a graphics data generating unit that generates graphics data to        display graphics on an image; and    -   an image data processing unit that uses the image data obtained        and the smoothed disparity information and the generated        graphics data, appends disparity corresponding to the display        position of the graphics for each picture to the graphics that        overlay a left eye image and right eye image, and obtains left        eye image data onto which the graphics have been overlaid and        right eye image data onto which the graphics have been overlaid.

According to the present technology, a container of a predeterminedformat including video stream is received by an image data receivingunit. The video stream herein is obtained by the left eye image data andright eye image data that configure the stereoscopic image having beenencoded. Also, representative disparity information that is the otherdisparity information as to one of the left eye image and the right eyeimage obtained corresponding to each of a predetermined number ofpartition region of the picture display screen, and that is in eachpartition region corresponding to the partition pattern of the picturedisplay screen, is inserted into the video stream herein for eachpicture of the image data.

Left eye image data and right eye image data is obtained from the videostream included in the container, while representative disparityinformation for each partition region of each picture of the image datahere is obtained, by the information obtaining unit. Also, processingfor smoothing the representative disparity information for eachpartition basin of each picture in the temporal axis direction isperformed by an information smoothing unit. Also, graphics data todisplay graphics on an image is generated by a graphics data generatingunit. The graphics here are graphic such as OSD or applications, or EPGinformation indicating service content, for example.

The obtained image data and the smoothed disparity information and thegenerated graphics data are used, and data of the left eye image andright eye image where graphics are overlaid is obtained by the imagedata processing unit. In this case, disparity corresponding to thedisplay position of the graphics herein is appended to each picture inthe graphics that are overlaid onto the left eye image and right eyeimage, whereby data for the left eye image where graphics are overlaidand data for the right eye image where graphics are overlaid areobtained. For example, in the image data processing unit, the disparityinformation selected from the disparity information of a predeterminednumber of disparity regions corresponding to the display position of thegraphics, for example optimal disparity information such as a minimumvalue, is used, and appending the disparity to the graphics herein isperformed.

Thus, according to the present technology, depth control of the graphicsthat are overlaid onto the stereoscopic image and displayed isperformed, based on the disparity information inserted into the videostream that is transmitted from the transmitting side. In this case, therepresentative disparity information of each partition region obtainedfor each picture of the image data is inserted into the video stream,and depth control of the graphics can be favorable performed withpicture (frame) precision. Also, in this case, smoothing processing inthe temporal axis direction is performed and used as to therepresentative disparity information for each partition basin of eachpicture, whereby even as to sudden changes in the disparity information,the occurrence of viewer discomfort can be reduced.

Advantageous Effects of Invention

According to the present invention, transmission of related informationof image data such as disparity information or the like can be performedfavorably.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animage transmitting/receiving system according to an embodiment.

FIG. 2 is a diagram illustrating an example of disparity information(disparity vectors) for each block.

FIG. 3 is a diagram to describe an example of a generating method ofdisparity information in increments of blocks.

FIG. 4 is a diagram to describe an example of downsizing processingwhich is in order to obtain disparity information of predetermineddivided regions from the disparity information for each block.

FIG. 5 is a diagram to describe a picture display screen being dividedso that encoded block borders are not straddled.

FIG. 6 is a diagram schematically illustrating an example of transitionof disparity information of each partition region for each picture.

FIG. 7 is a diagram to describe the timing for inserting disparityinformation obtained for each picture of the image data into the videostream.

FIG. 8 is a block diagram illustrating a configuration example oftransmission data generating unit to generate a transport stream at abroadcast station.

FIG. 9 is a diagram illustrating a configuration example of a transportstream.

FIG. 10 is a diagram illustrating a configuration example (Syntax) andprimary prescribed content (semantics) of an AVC video descriptor.

FIG. 11 is a diagram illustrating a configuration example (Syntax) andprimary prescribed content (semantics) of an MVC extension descriptor.

FIG. 12 is a diagram illustrating a configuration example (Syntax) andprimary prescribed content (semantics) of a graphics depth infodescriptor (graphics_depth_info_descriptor).

FIG. 13 illustrates an example of a head access unit of a GOP and anaccess unit other than a head of a GOP, in the case that the encodingformat is AVC.

FIG. 14 is a diagram illustrating a configuration example (Syntax) of“depth_information SEI message” and configuration example (Syntax) of“depth_information_data( )”.

FIG. 15 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case of inserting disparity informationfor each picture in picture increments.

FIG. 16 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case of inserting disparity informationfor each picture in picture increments.

FIG. 17 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case of inserting disparity informationfor each picture in picture increments.

FIG. 18 is a diagram illustrating content (Semantics) of primaryinformation in the configuration example (Syntax) of “depth_information()”.

FIG. 19 is a diagram illustrating a partitioning example of a picturedisplay screen.

FIG. 20 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case that disparity information for eachpicture is encoded for multiple pictures together.

FIG. 21 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case that disparity information for eachpicture is encoded for multiple pictures together.

FIG. 22 is a diagram illustrating a configuration example (Syntax) of“depth_information( )” in the case that disparity information for eachpicture is encoded for multiple pictures together.

FIG. 23 is a diagram illustrating a configuration example (Syntax) of“user_data( )” and a configuration example (Syntax) of “depthinformation_data( )”.

FIG. 24 is a diagram illustrating a concept of depth control of graphicsby disparity information.

FIG. 25 is a diagram illustrating that disparity information is obtainedsequentially at picture timing of the image data, in the case thatdisparity information is inserted into the video stream in pictureincrements.

FIG. 26 is a diagram illustrating that disparity information for eachpicture within the GOP is obtained together at the head timing of theGOP of the image data, in the case that disparity information isinserted into the video stream in GOP increments.

FIG. 27 is a diagram illustrating a display example of a subtitle on animage and OSD graphics.

FIG. 28 is a block diagram illustrating a configuration example of aset-top box.

FIG. 29 is a diagram to describe checking of disparity angles andreconfiguration of image data.

FIG. 30 is a block diagram illustrating a configuration example of adisparity information control unit.

FIG. 31 is a block diagram to describe the control of the depth controlunit.

FIG. 32 is a diagram to describe an example of a filtering processingthat smoothes in the temporal direction.

FIG. 33 is diagram to describe another example of a filtering processingthat smoothes in the temporal direction.

FIG. 34 is a flowchart (1/2) illustrating an example of a controlprocessing sequence of the depth control unit.

FIG. 35 is a flowchart (2/2) illustrating an example of a controlprocessing sequence of the depth control unit.

FIG. 36 is a diagram illustrating a depth control example of graphics ina set-top box.

FIG. 37 is a diagram illustrating another depth control example ofgraphics in a set-top box.

FIG. 38 is a block diagram illustrating a configuration example of atelevision receiving device (HDMI input system).

FIG. 39 is a block diagram to describe the control of the depth controlunit.

FIG. 40 is a flowchart illustrating an example of the sequence of thecontrol processing of the depth control unit.

FIG. 41 is a diagram illustrating a depth control example of graphics ona television receiving device.

FIG. 42 is a block diagram illustrating a configuration example of anHDMI transmitting unit of a source device and an HDMI receiving unit ofa sink device.

FIG. 43 is a diagram illustrating a configuration example of TMDStransmission data (in the case that image data having 1920 horizontalpixels by 1080 vertical lines is transmitted).

FIG. 44 is a diagram illustrating a packet configuration example of anHDMI Vendor Specific InfoFrame, in the case that HDMI Vendor SpecificInfoFrame is used in the transmission of disparity information.

FIG. 45 is a diagram illustrating content of primary information in thepacket configuration example of the HDMI Vendor Specific InfoFrame.

FIG. 46 is a diagram illustrating a configuration example of VS_Info, inthe case that the mode is for single picture, and the partition regionis “16”.

FIG. 47 is a diagram schematically illustrating a case in which pictureincrement receiving and single picture mode transmitting are performed.

FIG. 48 is a diagram schematically illustrating a case in which pictureincrement receiving and double picture mode transmitting are performed.

FIG. 49 is a diagram schematically illustrating a case in which GOPincrement (multiple picture increment) receiving and single picture modetransmission are performed.

FIG. 50 is a diagram schematically illustrating a case in which GOPincrement (multiple picture increment) receiving and double picture modetransmission are performed.

FIG. 51 is a diagram illustrating another packet configuration exampleof VS_Info (HDMI Vendor Specific InfoFrame).

FIG. 52 is a diagram illustrating another packet configuration exampleof VS_Info (HDMI Vendor Specific InfoFrame).

FIG. 53 is a block diagram illustrating a configuration example toperform order determination of N picture and N+1 picture.

FIG. 54 is a diagram illustrating a time shift example of the disparityinformation (Disparity value) in the case where |D(N+1)−D(N)|≦Th.

FIG. 55 is a diagram illustrating a time shift example of the disparityinformation (Disparity value) in the case where |D(N+1)−D(N)|>Th.

FIG. 56 is a diagram illustrating a configuration example of a packetheader of a 3D displaying support packet as the data packet that isnewly defined.

FIG. 57 is a diagram illustrating a configuration example of packetcontents.

FIG. 58 is a diagram illustrating another configuration example ofpacket contents.

FIG. 59 is a diagram illustrating another configuration example ofpacket contents.

FIG. 60 is a diagram illustrating a 3D video format of a frame packingmethod which is one TMDS transmission data configuration of thestereoscopic image data.

FIG. 61 is a diagram illustrating an example of a packet configurationof the HDMI Vendor Specific InfoFrame, in the case of using an activespace region.

FIG. 62 is a block diagram illustrating another configuration example ofan image transmitting/receiving system.

FIG. 63 is a block diagram illustrating a configuration example of thetelevision receiving device.

FIG. 64 is a diagram illustrating the relation between the displaypositions of the left and right images of an object on a screen and theplaying position of the stereoscopic image thereof, in a stereoscopicimage display that uses binocular disparity.

DESCRIPTION OF EMBODIMENTS

Embodiments to realize the invention (hereinafter called “embodiments”)will be described below. Note that descriptions will be given in theorder below.

1. Embodiment

2. Modifications

1. Embodiment Image Transmitting/Receiving System

FIG. 1 illustrates a configuration example of an imagetransmitting/receiving system 10 as an embodiment. The imagetransmitting/receiving system 10 has a broadcast station 100, a set-topbox (STB) 200, and a television receiving device 300 serving as amonitor. The set-top box 200 and television receiving device (TV) 300are connected via an HDMI (High Definition Multimedia Interface) cable400.

“Description of Broadcast Station”

The broadcast station 100 transmits a transport stream TS as a containeron broadcast waves. The video stream obtained by left eye image data andright eye image data which make up a stereoscopic image having beenencoded is included in this transport stream TS. For example, the lefteye image data and right eye image data are transmitted in one videostream. In this case, for example, the left eye image data and right eyeimage data are subjected to interleaving processing, configured as imagedata of a side-by-side format or top-and-bottom format, and included inone video stream.

Alternatively, for example, the left eye image data and right eye imagedata are each transmitted in separate video streams. In this case, forexample, the left eye image data is included in a MVC base view stream,and the right eye image data is included in a MVC non-base view stream.

Other disparity information (Disparity data) as to one of the left eyeimage and right eye image, obtained for each predetermined picture ofthe image data, is inserted into the video stream. In this case,obtaining the disparity information presupposes that at least thedisparity information is performed with pictures that should betransmitted.

Now, the disparity information for each picture is made ofrepresentative disparity information in the predetermined regions of thepicture display screen. According to this embodiment, first disparityinformation and second disparity information are included as thisrepresentative disparity information, but only the first disparityinformation being included may also be conceived. The first disparityinformation is disparity information corresponding to the nearest objectplaying position in the predetermined region of the picture displayscreen. Also, the second disparity information is disparity informationcorresponding to the farthest object playing position in thepredetermined region of the picture display screen.

If the screen position is at disparity zero, in the case that the objectplaying position is on the near side of the screen, the disparityinformation herein can be obtained with a negative value (see DPa inFIG. 64). On the other hand, in the case that the object playingposition is on the far side of the screen, the disparity informationherein can be obtained with a positive value (see DPc in FIG. 64).Therefore, as the first disparity information, for example, of thedisparity information in the predetermined regions, the minimum valuedisparity information may be used. Also, as the second disparityinformation, for example, of the disparity information in thepredetermined regions, the maximum value disparity information may beused.

In this case, as the first disparity information and second disparityinformation, for example, the following obtain methods may be conceived.

(1) First disparity information in each partition region obtained bypartitioning the picture display screen into a plurality is obtained,and second disparity information of the entire picture display screen isobtained, based on partition pattern information.

(2) First disparity information and second disparity information in eachpartition region obtained by partitioning the picture display screeninto a plurality are obtained, based on partition pattern information.

(3) First disparity information in each partition region obtained bypartitioning the picture display screen with first partitioninginformation is obtained, and second disparity information in eachpartition region obtained by partitioning the picture display screenwith second partitioning information is obtained, based on partitionpattern information.

(4) First disparity information in the entire picture display screen isobtained, and second disparity information in the entire picture displayscreen is obtained, based on partition pattern information.

The disparity information in the entire picture display screen or ineach partition region is obtained by the disparity information for eachblock being subjected to downsizing processing. FIG. 2 illustrates andexample of disparity information (disparity vector) for each block.

FIG. 3 illustrates an example of a generating method for disparityinformation in block increments. This example is an example to finddisparity information that indicates a right eye view (Right-View) froma left eye view (Left-View). In this case, a pixel block (disparitydetection block) such as 4*4, 8*8, or 16*16, for example, is provided tothe picture of the left eye view.

As illustrated in the diagram, the picture of the left eye view is setas a detection image, the picture of the right eye view is set as areference image, and block searching is performed for the picture of theright eye view so that the sum of absolute differences between pixels isthe smallest, for each block of the picture of the left eye view,whereby disparity data is found.

That is to say, the disparity information DPn of the N'th block is foundby block searching so that the difference absolute value sum in the N'thblock herein is minimum, as shown in the Expression (1) below, forexample. Note that in Expression (1) herein, Dj indicates the pixelvalue in the picture of the right eye view, and Di indicates the pixelvalue in the picture of the left eye view.DPn=min(Σabs(differ(Dj−Di)))  (1)

FIG. 4 illustrates an example of downsizing processing. FIG. 4(a)illustrates disparity information for each block obtained as describedabove. Based on this disparity information for each block, firstdisparity information and second disparity information for each group(Group of Block) is found, as illustrated in FIG. 4(b). A group is ahierarchical level above a block, and is obtained by grouping multipleadjacent blocks. In the example in FIG. 4(b), each group is made up offour blocks bundled together with a broken-line frame. Also, the firstdisparity information of each group is obtained by the disparityinformation of the minimum value being selected from the disparityinformation of all of the blocks within the group thereof, for example.Also, the second disparity information of each group is obtained by thedisparity information of the maximum value being selected from thedisparity information of all of the blocks within the group thereof, forexample.

Next, based on the disparity vectors for each group, the first disparityinformation and second disparity information for each partition isfound, as illustrated in FIG. 4(c). A partition is a hierarchical levelabove a group, and is obtained by grouping together multiple adjacentgroups. In the example in FIG. 4(c), each partition is made up of twogroups bundled together with a broken-line frame. Also, the firstdisparity information of each partition is obtained by the minimum valuefirst disparity information being selected from the first disparityinformation of the entire group within the partition thereof, forexample. Also, the second disparity information of each partition isobtained by the maximum value second disparity information beingselected from the second disparity information of the entire groupwithin the partition thereof, for example.

Next, based on this disparity information for each partition, the firstdisparity information and second disparity information for the entirepicture (entire picture display screen), which is the uppermosthierarchical level, is found, as illustrated in FIG. 4(d). In theexample in FIG. 4(d), four partitions bundled together with abroken-line frame are included in the entire picture. Also, the firstdisparity information of the entire picture is obtained by the minimumvalue first disparity information being selected from the firstdisparity information of all of the partitions included in the entirepicture, for example. Also, the second disparity information of theentire picture is obtained by the maximum value second disparityinformation being selected from the second disparity information of allof the partitions included in the entire picture, for example.

The picture display screen is partitioned based on the partitioninformation, and the disparity information for each partition region isobtained as described above. In this case, the picture display screen ispartitioned so that the encoded block borders are not straddled. FIG. 5illustrates a partition detail example of a picture display screen. Thisexample is an example of a 1920*1080 pixel format, and is an examplewhere partitions are in two each horizontally and vertically, wherebyfour partition regions of Partition A, Partition B, Partition C, andPartition D are obtained. On the transmitting side, encoding isperformed for each block of 16×16, whereby eight lines made up of blankdata are appended, and encoding is performed as image data of 1920pixels 1088 lines. Now, in the vertical direction, partitioning is doneinto two, based on the 1088 lines.

As described above, the disparity information (first disparityinformation, second disparity information) of the entire picture displayscreen of the partition regions obtained for each predetermined picture(frame) of the image data is inserted in the video stream. FIG. 6schematically illustrates a shifting example of disparity information ofeach partition region. This example is an example where partitions arein four each horizontally and vertically, whereby sixteen partition ofPartition 0 through Partition 15 exist. In this example, forsimplification of the diagram, only the shifting of the disparityinformation D0, D3, D9, and D15 of Partition 0, Partition 3, Partition9, and Partition 15 are illustrated. There are cases where the eachvalue of disparity information may change with time, (D0, D3, D9) andcases where the values are fixed (D15).

The disparity information obtained for each predetermined picture of theimage data is inserted into the video stream in picture increments orGOP increments or the like. FIG. 7(a) illustrates an example ofsynchronizing with the picture encoding, i.e. an example of insertingdisparity information into the video stream in picture increments. Inthis example, delays in the event of transmitting the image data may befewer, so is appropriate to live broadcasting that transmits image dataimaged by a camera.

FIG. 7(b) illustrates an example of synchronizing with an I-picture(Intra picture) of the encoded video or a GOP (Group Of Pictures), i.e.an example of inserting the disparity information into the video streamin GOP increments. In this example, delays in the event of transmittingthe image data may increase as compared to the example in FIG. 7(a), butdisparity information for multiple pictures (frames) can be transmittedtogether, whereby the number of processing times to obtain the disparityinformation on the receiving side can be reduced. FIG. 7(c) illustratesan example of synchronizing with a video scene, i.e. an example ofinserting the disparity information into the video stream in sceneincrements. Note that FIGS. 7(a) through (c) are examples, and insertingin other increments may also be conceived.

Also, identification information to identify whether or not disparityinformation (first disparity information, second disparity information)has been inserted in the video stream may be inserted into a layer ofthe transport stream TS. This identification information is insertedbeneath a program map table (PMT) included in the transport stream TS orbeneath an event information table (EIT), for example. With thisidentification information, determination can be made readily on thereceiving side as to whether or not disparity information has beeninserted into the video stream. Details of this identificationinformation will be described later.

“Configuration Example of Transmission Data Generating Unit”

FIG. 8 illustrates a configuration example of a transmission datagenerating unit 110 that generates the above-described transport streamTS at the broadcast station 100. This transmission data generating unit110 has image data output units 111L and 111R, scalers 112L and 112R,video encoder 113, multiplexer 114, and disparity data generating unit115. Also, this transmission data generating unit 110 has a subtitledata output unit 116, subtitle encoder 117, audio data output unit 118,audio encoder 119, and partition pattern selecting unit 120.

The image data output units 111L and 111R output left eye image data VLand right eye image data VR, respectively, that make up the stereoscopicimage. The image data output units 111L and 111R are made up of a camerathat images a subject and outputs image data or an image data readoutunit that reads out image data and outputs from a storage medium, or thelike, for example. The image data VL and VR are image data of a sizethat is full-HD of 1920*1080, for example.

The scalers 112L and 112R perform scaling processing in the horizontaldirection and vertical direction as needed, as to the image data VL andVR, respectively. For example, in order to transmit the image data VLand VR in one video stream, in the case of configuring the image datawith a side-by-side format or top-and-bottom format, the image data isscaled down to ½ in the horizontal direction or the vertical direction,and is output. Also, for example, in the case of transmitting the imagedata VL and VR in individual video streams such as an MVC base viewstream and non-base view stream, respectively, scaling processing is notperformed, and the image data VL and VR is output without change.

The video encoder 113 performs encoding such as MPEG4-AVC (MVC),MPEG2video, or HEVC (High Efficiency Video Coding) or the like, forexample, as to the left eye image data and right eye image data outputfrom the scalers 112L and 112R, thereby obtaining encoded video data.Also, this video encoder 113 generates a video stream including theencoded data herein with a stream formatter (unshown) that is providedat a later stage. In this case, the video encoder 113 generates one ortwo video streams (video elementary streams) that include an encodedvideo stream of the left eye image data and right eye image data.

The disparity data generating unit 115 generates disparity informationfor each picture (frame) based on the left eye image data VL and righteye image data VR that is output from the image data output units 111Land 111R. In this case, the disparity data generating unit 115 performsprocessing to obtain the first disparity information and seconddisparity information, based on information of the partition patternselected with the partition pattern selecting unit 120 according tooperations by the user, for example.

First, the disparity data generating unit 115 obtains disparityinformation for each block as described above, for each picture. Notethat in the case that the image data output units 111L and 111R areimage data readout units that have a storage medium, a configuration canbe conceived wherein the disparity data generating unit 115 reads outand obtains the disparity information for each block together with theimage data from the storage medium. Also, a method may be conceived touse the results from block matching that is performed between the righteye image data and left eye image data in the video encoder 113, and todetect disparity information.

Subsequently, the disparity information generating unit 115 performsdownsizing processing as to the disparity information for each block,and generates the first disparity information and second disparityinformation of the entire picture display screen or each partitionregion obtained by partitioning the picture display screen. In thisevent, the partition pattern information described above is used. Withthis partition pattern information, information is provided to thedisparity data generating unit 115, such as generating disparityinformation over the entire picture display screen, or generatingdisparity information in each partition region obtained by partitioningthe picture display screen by a predetermined number, or the like.

The video encoder 113 inserts the first disparity information and seconddisparity information for each picture generated by the disparity datagenerating unit 115 into the video stream. In this case, for example,the disparity information for each picture is inserted in the videostream in picture increments or in GOP increments (see FIG. 7). Notethat in the case that the left eye image data and right eye image dataare each transmitted with separate video data, insertion may be intojust one of the video streams.

The subtitle data output unit 116 outputs the data of subtitles thatoverlay the image. The subtitle data output unit 116 is made up of apersonal computer or the like, for example. The subtitle encoder 117generates the subtitle stream (subtitle elementary stream) that includesthe subtitle data output from the subtitle data output unit 116. Notethat the subtitle encoder 117 references the disparity information foreach block that is generated by the disparity data generating unit 115,and appends disparity information corresponding to the display positionof the subtitle to the subtitle data. That is to say, subtitle dataincluded in the subtitle stream holds disparity informationcorresponding to the display position of the subtitle.

The audio data output unit 118 outputs audio data corresponding to theimage data. This audio data output unit 118 is made up of a microphoneor an audio data readout unit or the like that reads and outputs audiodata from a storage medium, for example. The audio encoder 119 performsencoding such as MPEG-2Audio, AAC, or the like as to the audio dataoutput from the audio data output unit 118, and generates an audiostream (audio elementary stream).

The multiplexer 114 subjects each elementary stream generated by thevideo encoder 113, subtitle encoder 117, and audio encoder 119 to PESpacketizing, and generates a transport stream TS. In this case, a PTS(Presentation Time Stamp) is inserted into each the header of each PES(Packetized Elementary Stream) packet for synchronized playing on thereceiving side.

The multiplexer 114 inserts the above-described identificationinformation in a layer of the transport stream TS. This identificationinformation is information to identify whether or not disparityinformation (first disparity information, second disparity information)has been inserted in the video stream. This identification informationis inserted beneath a program map table (PMT) included in the transportstream TS or beneath an event information table (EIT) or the like, forexample.

Operations of the transmission data generating unit 110 illustrated inFIG. 8 will be briefly described. The left eye image data VL and righteye image data VR that configure the stereoscopic image output from theimage data output units 111L and 111R are supplied to the scalers 112Land 112R, respectively. Scaling processing in the horizontal directionand vertical direction is performed as needed as to the image data VLand VR, with the scalers 112L and 112R, respectively. The left eye imagedata and right eye image data output from the scalers 112L and 112R aresupplied to the video encoder 113.

With the video encoder 113, encoding such as MPEG4-AVC (MVC),MPEG2video, and HEVC or the like, for example, is performed as to theleft eye image data and right eye image data, and encoded video data isobtained. Also, with this video encoder 113, a video stream thatincludes this encoded data is generated by a stream formatter which willbe provided at a later stage. In this case, one or two video streamsthat include the encoded video data of the left eye image data and righteye image data are generated.

Also, the left eye image data VL and right eye image data VR thatconfigure the stereoscopic image output from the image data output units111L and 111R is supplied to the disparity data generating unit 115.With this disparity data generating unit 115, disparity information foreach block is obtained for each picture. Also, with this disparity datagenerating unit 115, further, downsizing processing is performed as tothe disparity information for each block, and first disparityinformation and second disparity information is generated in eachpartition region obtained by partitioning the entire picture displayscreen or the picture display screen, based on the partition patterninformation.

The first disparity information and second disparity information foreach picture that is generated by the disparity data generating unit 115is supplied to the video encoder 113. With the video encoder 113, thefirst disparity information and second disparity information for eachpicture is inserted into the video stream in picture increments or inGOP increments.

Also, the data of the subtitles that overlay onto the image is output bythe subtitle data output unit 116. This subtitle data is supplied to thesubtitle encoder 117. A subtitle stream that includes the subtitle datais generated by the subtitle encoder 117. In this case, the disparityinformation for each block generated with the disparity data generatingunit 115 is referenced at the subtitle encoder 117, and disparityinformation corresponding to the display position is appended to thesubtitle data.

Also, audio data corresponding to the image data is output by the audiodata output unit 118. This audio data is supplied to the audio encoder119. Encoding such as MPEG-2Audio, AAC, and the like is performed as tothe audio data with this audio encoder 119, and the audio stream isgenerated.

The video stream obtained with the video encoder 113, the subtitlestream obtained with the subtitle encoder 117, and the audio streamobtained with the audio encoder 119 are each supplied to the multiplexer114. The elementary stream supplied from each encoder is subjected toPES packetizing and multiplexed by the multiplexer 114, and a transportstream TS is generated. In this case, PTS is inserted into each of thePES headers for synchronized playing on the receiving side. Also, in themultiplexer 114, identification information to identify whether or notdisparity information has been inserted into the video stream isinserted beneath a PMT or beneath an EIT, or the like.

[Identification Information, Configuration of Disparity Information, TSConfiguration]

FIG. 9 illustrates a configuration example of a transport stream TS. Inthis configuration example, an example is illustrated where left eyeimage data and right eye image data are each transmitted in separatevideo streams. That is to say, a PES packet “video PES1” of a videostream in which left eye image data is encoded and a PES packet “videoPES2” of a video stream in which right eye image data is encoded areincluded. Also, a PES packet “subtitle PES3” of a subtitle in whichsubtitle data (including disparity information) is encoded and a PESpacket “audio PES4” of an audio stream in which audio data is encoded,is included in this configuration example.

Depth information/SEI (depth_information( )) that includes the firstdisparity information and second disparity information for each pictureis inserted in a user data region of the video stream. For example, inthe case that disparity information for each picture is inserted inpicture increments, this depth information/SEI is inserted into the userdata region of each picture in the video stream. Also, for example, inthe case that the disparity information for each picture is inserted inGOP increments, this depth information/SEI is inserted into the userdata region of the picture that corresponds to the head of the GOP ofthe video stream or the position where the sequence parameterinformation is inserted. Note that in the illustration of thisconfiguration example, depth information/SEI is inserted into both oftwo video streams, but insertion may be into only one of the videostreams.

A PMT (Program Map Table) is included as a PSI (Program SpecificInformation) in the transport stream TS. This PSI is information thatindicates the program to which each elementary stream included in thetransport stream TS belongs. Also, an EIT (Event Information Table)serving as an SI (Serviced Information) that performs managing in eventincrements is included in the transport stream TS.

An elementary loop having information that is related to each elementarystream exists beneath the PMT. Information such as a packet identifier(PID) for each stream is disposed on this elementary loop, while adescriptor that describes information related to the elementary streamis also disposed.

In the case of inserting identification information illustrating whetheror not disparity information (first disparity information, seconddisparity information) is inserted into the above-described videostream, this is described in a descriptor that is inserted beneath thevideo elementary loop of the program map table, for example. Thisdescriptor is, for example, an existing AVC video descriptor or MVCextension descriptor (MVC extension descriptor), or is a newly defineddepth info descriptor (Depth_info_descriptor). Note that regarding thedepth info descriptor, inserting beneath the EIT as described withbroken lines in the diagram may also be conceived.

FIG. 10(a) illustrates a configuration example (Syntax) of an AVC videodescriptor in which identification information is described. Thisdescriptor can be applied in the case that the video is in MPEG4-AVCFrame compatible format. This descriptor itself is already in thespecifications of H.264/AVC. Now, one-bit flag information of“depth_info_not_existed_flag[0]” is newly defined in this descriptor.

This flag information indicates whether or not the depth information/SEI(depth_informtion_sei( )) that includes disparity information for eachpicture is inserted into the corresponding video stream, as illustratedin the specification content (semantics) in FIG. 10(b). When this flaginformation is “0”, this indicates insertion. On the other hand, whenthis flag information is “1”, this indicates no insertion.

FIG. 11(a) illustrates a configuration example (Syntax) of the MVCextension descriptor in which identification information is described.This descriptor can be applied in the case that the video is inMPEG4-AVC Annex HMVC format. This descriptor itself is already in thespecifications of H.264/AVC. Now, one-bit flag information of“depth_info_not_existed_flag” is newly defined in this descriptor.

This flag information indicates whether or not depth information/SEI(depth_information( )) that includes disparity information for eachpicture is inserted in the corresponding video stream, as illustrated inthe specification content (semantics) in FIG. 11(b). When this flaginformation is “0”, this indicates insertion. On the other hand, whenthis flag information is “1”, this indicates no insertion.

FIG. 12(a) illustrates a configuration example (Syntax) of the depthinfo descriptor (depth_info_descriptor). An 8-bit field of“descriptor_tag” indicates that this descriptor is a“depth_info_descriptor”. Numbers of data bytes thereafter are indicatedin the 8-bit field of “descriptor_length”. Also, 1-bit flag informationof “depth_info_not-existed_flag” is described in this descriptor.

This flag information indicates whether or not depth information/SEI(depth_information( )) that includes disparity information for eachpicture is inserted in the corresponding video stream, as illustrated inthe specification content (semantics) in FIG. 12(b). When this flaginformation is “0”, this indicates insertion. On the other hand, whenthis flag information is “1”, this indicates no insertion.

Next, description will be given for a case where depth information/SEI(depth_information( )) that includes disparity information for eachpicture is inserted into the user data region of the video stream.

For example, in the case that the encoding format is AVC,“depth_information( )” is inserted as “depth_information SEI message” inthe “SEIs” portion of an access unit. FIG. 13(a) indicates an accessunit of the head of a GOP (Group Of Pictures), and FIG. 13(b) indicatesan access unit at other than the head of the GOP. In the case thatdisparity information for each picture is inserted in GOP increments,the “depth_informaiton SEI message” is inserted only into the accessunit at the head of the GOP.

FIG. 14(a) illustrates a configuration example (Syntax) of“depth_informaiton SEI message”. “uuid_iso_iec_11578” has a UUID valueexpressed as “ISO/IEC 11578:1996 AnnexA.”. The “depth_information_data()” is inserted in the “user_data_payload_byte” field. FIG. 14(b)illustrates a configuration example (Syntax) of “depth_informaiton_data()”. Depth information/SEI “depth_information_data( )” is insertedtherein. “userdata_id” is an identifier of the “depth_information( )”that is expressed in 16 bits with no sign.

FIG. 15(a), FIG. 16, and FIG. 17 each illustrate a configuration example(Syntax) of “depth_information( )” in the case of inserting disparityinformation for each picture in picture increments. FIG. 18 illustratescontent (Semantics) of primary information of these configurationexamples.

FIG. 15(a) illustrates a configuration example (Syntax) that correspondsto an obtaining method of “obtain first disparity information in eachpartition region obtained by partitioning a picture display screen intoa plurality, based on partition pattern information, and obtain seconddisparity information over the entire picture display screen” in (1)described above.

A 3-bit field of “partition_type” indicates a partition type of thepicture display screen. “000” indicates partition type “type000”, asillustrated in FIG. 19(a). “001” indicates partition type “type001”, asillustrated in FIG. 19(b). “010” indicates partition type “type010”, asillustrated in FIG. 19(c). “011” indicates partition type “type011”, asillustrated in FIG. 19(d). “100” indicates partition type “type100”, asillustrated in FIG. 19(e). “101” indicates partition type “type101”, asillustrated in FIG. 19(f).

A 4-bit field of “partition_count” indicates the total number ofpartition regions, which is a value that depends on the above-described“partition_type”. For example, when “partition_type=000”, the totalnumber of partition regions is “1”, as indicated in FIG. 19(a). Also,for example, when “partition_type=001”, the total number of partitionregions is “4”, as indicated in FIG. 19(b). Also, for example, when“partition_type=010”, the total number of partition regions is “8”, asindicated in FIG. 19(c).

An 8-bit field of “max_disparity_in_picture” indicates the seconddisparity information of the overall picture display screen, i.e. themaximum disparity information (disparity value) over the entire picture.An 8-bit field of “min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e. the minimumdisparity information (disparity value) in each partition region.

Now, “min_disparity_in_partition” which is the first disparityinformation and “max_disparity_in_picture” which is the second disparityinformation are set as absolute value data. “min_disparity_in_partition”can be restricted to the forward side from the monitor position, wherebyeven without indicating sign bits, on the receiving side, the disparityinformation value (absolute value) to be transmitted can be converted toa negative value and used, as illustrated in FIG. 15(b). Similarly,“max_disparity_in_picture” can be restricted to the far side from themonitor position, whereby even without indicating sign bits, on thereceiving side, the disparity information value (absolute value) to betransmitted can be converted to a positive value and used, asillustrated in FIG. 15(b).

Thus, the range of values of the disparity information expressed with 8bits is (−255, +255), and necessary disparity expressions can be usedwith 8 bits, even with a superhigh quality image having a resolution atthe level of 4,000 horizontal pixels, for example. Also, as to aconventional resolution at the level of 2,000 horizontal pixels, the 8bits can be divided into “7 bits” plus “1 bit”, where a disparityexpression of (−127.5, +127.5) can be used. In this case, the decimalportions express a half-pixel, and by appropriately interpolating thegraphics object to be overlaid, a smoother automatic update in the depthdirection can be realized.

FIG. 16 illustrates a configuration example (Syntax) that corresponds toan obtaining method of “obtain first disparity information and seconddisparity information in each partition region obtained by partitioninga picture display screen into a plurality, based on partition patterninformation” in (2) described above. Portions in FIG. 16 herein thatcorrespond to FIG. 15 will have the descriptions thereof omitted asappropriate.

A 3-bit field of “partition_type” indicates a partition type of thepicture display screen. A 4-bit field of “partition_count” indicates thetotal number of partition regions, which is a value that depends on theabove-described “partition_type”. An 8-bit field of“max_disparity_in_partition” indicates the second disparity informationof each partition region, i.e. the maximum disparity information(disparity value) in each partition region. An 8-bit field of“min_disparity_in_partition” indicates the first disparity informationof each partition region, i.e. the minimum disparity information(disparity value) in each partition region. Now,“min_disparity_in_partition” which is the first disparity informationand “max_disparity_in_partition” which is the second disparityinformation are set as absolute value data.

FIG. 17 illustrates a configuration example (Syntax) that corresponds toan obtaining method of “obtain first disparity information in eachpartition region obtained by partitioning a picture display screen withfirst partition information, and obtain second disparity information ineach partition region obtained by partitioning a picture display screenwith second partition information based on partition patterninformation” in (3) described above. Portions in FIG. 17 herein thatcorrespond to FIG. 15 and FIG. 16 will have the descriptions thereofomitted as appropriate.

The 3-bit field of “min_partition_type” indicates a partition type ofpicture display screen related to obtaining the first disparityinformation. The 4-bit field of “min_partition_count” indicates thetotal number of partition regions in which first disparity informationis obtained, and is a value that depends on the above-described“min_partition_type”. The 3-bit field of “max_partition_type” indicatesa partition type of picture display screen relating to obtaining thesecond disparity information. The 4-bit field of “max_partition_count”indicates the total number of partition regions in which seconddisparity information is obtained, and is a value that depends on theabove-described “max_partition_type”.

The 8-bit field of “min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e. the minimumdisparity information (disparity value) in each partition region. An8-bit field of “max_disparity_in_partition” indicates the seconddisparity information of each partition region, i.e. the maximumdisparity information (disparity value) in each partition region. Now,“min_disparity_in_partition” which is the first disparity informationand “max_disparity_in_partition” which is the second disparityinformation are set as absolute value data.

FIG. 20, FIG. 21, and FIG. 22 each illustrate a configuration example(Syntax) of “depth_information( )” in the case of encoding multiplepictures as in the case of inserting the disparity information for eachpictures in GOP increments. Content (Semantics) of primary informationin these configuration examples are illustrated in FIG. 18.

FIG. 20 illustrates a configuration example (Syntax) that corresponds toan obtaining method of “obtain first disparity information in eachpartition region obtained by partitioning a picture display screen intoa plurality, based on partition pattern information, and obtain seconddisparity information over the entire picture display screen” in (1)described above. Portions in FIG. 20 herein that correspond to FIG. 15described above will have the descriptions thereof omitted asappropriate.

A 6-bit field of “picture_count” indicates the number of pictures. Firstdisparity information and second disparity information of the numbers ofpictures is included in this “depth_information( )”. A 4-bit field of“partition_count” indicates the total number of partition regions. An8-bit field of “max_disparity_in_picture” indicates the second disparityinformation of the entire picture display screen, i.e. the maximumdisparity information (disparity value) over the entire picture. An8-bit field of “min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e. the minimumdisparity information (disparity value) in each partition region.Detailed descriptions will be omitted, but other portions in theconfiguration example in FIG. 20 are similar to the configurationexample illustrated in FIG. 15.

FIG. 21 illustrates a configuration example (Syntax) that corresponds toan obtaining method of “obtain first disparity information and seconddisparity information in each partition region obtained by partitioninga picture display screen into a plurality, based on partition patterninformation” in (2) described above. Portions in FIG. 21 herein thatcorrespond to FIG. 16 and FIG. 20 will have the descriptions thereofomitted as appropriate.

The 6-bit field of “picture_count” indicates the number of pictures.First disparity information and second disparity information of thenumbers of pictures is included in this “depth_information( )”. The4-bit field of “partition_count” indicates the total number of partitionregions. The 8-bit field of “max_disparity_in_partition” indicates thesecond disparity information of each partition region, i.e. the maximumdisparity information (disparity value) in each partition region. The8-bit field of “min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e. the minimumdisparity information (disparity value) in each partition region.Detailed descriptions will be omitted, but other portions in theconfiguration example in FIG. 21 are similar to the configurationexample illustrated in FIG. 16.

FIG. 22 illustrates a configuration example (Syntax) that corresponds toan obtaining method of “obtain first disparity information in eachpartition region obtained by partitioning a picture display screen withfirst partition information, and obtain second disparity information ineach partition region obtained by partitioning a picture display screenwith second partition information, based on partition patterninformation” in (3) described above. Portions in FIG. 22 herein thatcorrespond to FIG. 17, FIG. 20, and FIG. 21 will have the descriptionsthereof omitted as appropriate.

The 6-bit field of “picture_count” indicates the number of pictures.First disparity information and second disparity information of thenumbers of pictures is included in this “depth_information( )”. The3-bit field of “min_partition_type” indicates a partition type ofpicture display screen related to obtaining the first disparityinformation. The 3-bit field of “max_partition_type” indicates apartition type of picture display screen relating to obtaining thesecond disparity information. The 4-bit field of “min_partition_count”indicates the total number of partition regions in which first disparityinformation is obtained, and the 4-bit field of “max_partition_count”indicates the total number of partition regions in which seconddisparity information is obtained.

The 8-bit field of “min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e. the minimumdisparity information (disparity value) in each partition region. The8-bit field of “max_disparity_in_partition” indicates the seconddisparity information of each partition region, i.e. the maximumdisparity information (disparity value) in each partition region.Detailed descriptions will be omitted, but other portions in theconfiguration example in FIG. 22 are similar to the configurationexample illustrated in FIG. 17.

Note that description is given above for a case in which the encodingformat is AVC. For example, in the case that the encoding format isMPEG2video, “depth_information( )” is inserted as user data “user_data()” in the user data region of the picture header portion. FIG. 23(a)illustrates a configuration example (Syntax) of “user_data( )”. The32-bit field of “user_data_start_code” is a starting code for user data(user_data), and is a fixed value of “0x000001B2”.

The 32-bit field that follows this starting code is an identifier thatidentifies the content of the user data. Here this is“depth_information_data_identifier”, and can identify that the user datais “depth_information_data”. As the main data after this identifier,“depth_information_data( )” is inserted. FIG. 23(b) illustrates aconfiguration example (Syntax) of “depth_information_data( )”.“depth_information( )” is inserted herein (see FIG. 15 through FIG. 17and FIG. 20 through FIG. 22).

Note that description is given of an insertion example of disparityinformation to the video stream in the case in which the encoding formatis AVC or MPEG2video. Detailed description will be omitted, butinsertion of disparity information into the video stream can beperformed with a similar configuration even with other similarconfigurations of encoding formats, e.g. HEVC.

“Description of Set-Top Box”

The set-top box 200 receives the transport stream TS that is transmittedon broadcast waves from the broadcast station 100. Also, the set-top box200 decodes the video stream included in this transport stream TS, andgenerates left eye image data and right eye image data that configuresthe stereoscopic image. Also, the set-top box 200 extracts the disparityinformation (first disparity information, second disparity information)for each picture of the image data, which is inserted in the videostream.

Based on the first disparity information and the second disparityinformation, the set-top box 200 checks whether or not the disparityangle corresponding to the nearest object playing position (disparityangle in the intersecting direction) and the disparity anglecorresponding to the farthest object playing position (disparity anglein the same-side direction) are within a predetermined range so as notto harm the health of the viewer. Also, in the case this is notcontained within a predetermined range, the set-top box 200 reconfiguresthe left eye image data and right eye image data so as to be contained,and corrects the first disparity information and second disparityinformation.

Also, in the case of overlaying and displaying the graphics (STBgraphics) on the image, the set-top box 200 uses the image data anddisparity information (first disparity information) and graphics data,and obtains data for the left eye image data and right eye image data onwhich graphics have been overlaid. In this case, the set-top box 200appends disparity, which corresponds to the display position of thesegraphics for each picture, to the graphics overlaid onto the left eyeimage and right eye image, and obtains left eye image data on whichgraphics are overlaid and right eye image data on which graphics areoverlaid.

By appending disparity to the graphics as described above, the graphics(STB graphics) overlaid and displayed on the stereoscopic image can bedisplayed nearer than the object of the stereoscopic image at thedisplay position thereof. Thus, in the case of overlaying and displayinggraphics such as an OSD or application or graphics such as EPG ofprogram information onto the image, consistency in perspective as toeach object within the image can be maintained.

FIG. 24 illustrates a concept of graphics depth control by the disparityinformation. In the case that the disparity information is a negativevalue, disparity is appended so that the graphics for the left eyedisplay shifts to the right side of the screen and the graphics for theright eye display shifts to the left side. In this case, the displayposition of the graphics is on the near side of the screen. Also, in thecase that the disparity information is a positive value, disparity isappended so that the graphics for the left eye display shifts to theleft side of the screen and the graphics for the right eye displayshifts to the right side. In this case, the display position of thegraphics is on the far side of the screen.

As described above, disparity information obtained for each picture ofthe image data is inserted into the video stream. Therefore, the set-topbox 200 can accurately perform graphics depth control with disparityinformation, using disparity information that matches the display timingof the graphics.

Also, FIG. 24 illustrates the disparity angle in the same-side direction(θ0-θ2) and the disparity angle in the intersecting direction (θ0-θ1). Acheck is performed at the set-top box 200 as to whether or not thesedisparity angles are within a predetermined range so as not to harm thehealth of the viewer, based on the first disparity information andsecond disparity information of each picture that are inserted in thevideo stream.

FIG. 25 is an example in the case of disparity information beinginserted in the video stream in picture increments, and at the set-topbox 200, disparity information is obtained sequentially with the picturetiming of the image data. Disparity information (first disparityinformation) matching the graphics display timing is used in the displayof graphics, and appropriate disparity is appended to the graphics.Also, FIG. 26 is an example in the case of disparity information beinginserted in the video stream in GOP increments, and at the set-top box200, disparity information (disparity information set) is obtainedtogether, for each picture within the GOP, at the head timing of the GOPof the image data. Disparity information (first disparity information)matching the graphics display timing is used in the display of graphics(STB graphics), and appropriate disparity is appended to the graphics.

The “Side View” in FIG. 27(a) illustrates a display example of thesubtitle on the image and the OSD graphic. This display example is anexample where subtitles and graphics are overlaid onto the image that ismade up of a background, mid-field object, and near-field object. The“Top View” in FIG. 27(b) illustrates perspectives of the background,mid-field object, near-field object, subtitle, and graphics. Theillustration shows that the subtitle and graphics are recognized asbeing nearer than objects corresponding to the display positions. Notethat while not illustrated in the diagram, in the case that the displaypositions of the subtitle and graphics are overlaid, appropriatedisparity is appended to the graphics so that the graphics arerecognized to be nearer than the subtitle, for example.

“Configuration Example of Set-Top Box”

FIG. 28 illustrates a configuration example of the set-top box 200. Theset-top box 200 has a container buffer 211, demultiplexer 212, codedbuffer 213, video decoder 214, decoded buffer 215, scaler 216, L/Rreconfiguring unit 263, and overlaying unit 217. Also, the set-top box200 has a disparity information buffer 218, disparity informationcontrol unit 262, set-top box (STB) graphics generating unit 219, depthcontrol unit 220, and graphics buffer 221.

Also, the set-top box 200 has a coded buffer 231, subtitle decoder 232,pixel buffer 233, subtitle disparity information buffer 234, andsubtitle display control unit 235. Further, the set-top box 200 has acoded buffer 241, audio decoder 242, audio buffer 243, channel mixingunit 244, and HDMI transmitting unit 251.

The container buffer 211 temporarily stores the transport stream TSreceived by an unshown digital tuner or the like. A video stream,subtitle stream, and audio stream are included in this transport streamTS. One or two video streams obtained by left eye image data and righteye image data having been encoded are included in a video stream.

For example, image data in a side-by-side format or top-and-bottomformat may be configured by left eye image data and right eye image dataand transmitted in one video stream. Also, for example, the left eyeimage data and right eye image data may each be transmitted in separatevideo streams, such as a MVC base view stream and non-base view stream.

The demultiplexer 212 extracts a stream for each of the video, subtitle,and audio from the transport stream TS that has been temporarily storedin the container buffer 211. Also, the demultiplexer 212 extractsidentification information (flag information of“graphics_depth_info_not_existed_flag[0]”) that indicates whether or notdisparity information is inserted in the video stream, and transmitsthis to an unshown control unit (CPU). When the identificationinformation indicates insertion of disparity information, the videodecoder 214 obtains disparity information (first disparity information,second disparity information) from the video stream, as described later,under the control of the control unit (CPU).

The coded buffer 213 temporarily stores the video stream extracted bythe demultiplexer 212. The video decoder 214 performs decodingprocessing on the video stream stored in the coded buffer 213, andobtains the left eye image data and right eye image data. Also, thevideo decoder 214 obtains the disparity information (first disparityinformation, second disparity information) for each picture of the imagedata inserted into the video stream. The decoded buffer 215 temporarilystores the left eye image data and right eye image data obtained withthe video decoder 214.

The disparity information buffer 218 temporarily stores the disparityinformation (first disparity information, second disparity information)for each picture of the image data obtained with the video decoder 214.The disparity information control unit 262 checks whether or not thedisparity angle corresponding to the nearest object playing position(disparity angle in the intersecting direction) and the disparity anglecorresponding to the farthest object playing position (disparity anglein the same-side direction) are within a predetermined range so as notto harm the health of the viewer, based on first disparity informationand second disparity information for each picture stored in thedisparity information buffer 218.

Also, in the case that the disparity angle is not contained within apredetermined range, the disparity information control unit 262instructs the L/R reconfiguration unit 263 to reconfigure the left eyeimage data and right eye image data so the disparity angle will becontained within a predetermined range, and corrects the first disparityinformation and second disparity information. Also, the disparityinformation control unit 262 corrects and outputs the first disparityinformation and/or the second disparity information to match thereconfigured left eye image data and right eye image data. Note that inthe case that the disparity angle is contained within a predeterminedrange, the disparity information control unit 262 outputs the left eyeimage data and right eye image data without change, without instructingthe L/R reconfiguring unit 263 to reconfigure the left eye image dataand right eye image data, and without correcting the first disparityinformation and second disparity information.

The upper portion of FIG. 29 illustrates an example of time shifting ofthe disparity angles on the near side and far sides in the receivingdata (left eye image data and right eye image data). A range d indicatesthe range of disparity angles that do not harm the health of the viewer.In this example, there is a segment in the disparity angle on the farside that is not contained in the range d.

The lower portion of FIG. 29 illustrates an example of time shifting ofthe disparity angles on the near side and far sides in the reconfigureddata (left eye image data and right eye image data). (a) is an examplein the case of limiting control being performed in the event that therange d has been exceeded. (b) is an example in the case that theoverall depth volume is reduced so that the range d is not exceeded.

FIG. 30 illustrates a configuration example of the disparity informationcontrol unit 262. The disparity information control unit 262 has amaximum value obtaining unit 271, minimum value obtaining unit 272,disparity angle checking unit 272, and disparity information correctingunit 274. The maximum value obtaining unit 271 selects the maximum valuefrom the received second disparity information (Max disparity), andsupplies this to the disparity angle checking unit 273. The receivedsecond disparity information is made up of one piece in the case ofbeing for an overall picture screen, and is multiple pieces in the caseof being for multiple partition regions. The minimum value obtainingunit 272 selects the minimum value from the received first disparityinformation (Min disparity), and supplies this to the disparity anglechecking unit 273. The received first disparity information is made upof one piece in the case of being for an overall picture screen, and ismultiple pieces in the case of being for multiple partition regions.

The disparity angle checking unit 273 checks whether or not thedisparity angle of the near side and the disparity angle of the far sideare contained within a range d (see FIG. 29) that are within apredetermined range so as not to harm the health of the viewer, based onfirst disparity information and second disparity information, andsupplies the results of checking to the disparity information correctingunit 274. When the disparity angle is not contained in the range d, thedisparity information correcting unit 274 instructs the L/Rreconfiguring unit 263 to reconfigure the received left eye image dataand right eye image data so that the disparity angles are contained inthe range d.

Also, the received first disparity information (Min disparity) andsecond disparity information (Max disparity) are supplied to thedisparity information correcting unit 274. The disparity informationcorrecting unit 274 performs correcting processing as to the firstdisparity information and second disparity information to match thereconfiguring instructions of the left eye image data and right eyeimage data described above, and outputs the first disparity information(Corrected Min disparity) and second disparity information (CorrectedMax disparity) after correction. Note that when the disparity angle iscontained in the range d, the disparity information correcting unit 274does not instruct the L/R reconfiguring unit 263 to reconfigure the lefteye image data and right eye image data, and outputs the first disparityinformation and second disparity information without correcting.

Returning to FIG. 28, the scaler 216 performs scaling processing in thehorizontal direction and vertical direction as to the left eye imagedata and right eye image data output from the decoded buffer 215, asneeded. For example, in the case that the left eye image data and righteye image data are transmitted in one video stream as image data of aside-by-side format or a top-and-bottom format, the data is scaled up todouble in the horizontal direction or the vertical direction, andoutput. Also, for example, in the case that the left eye image data andright eye image data is each transmitted in separate video streams, suchas a MVC base view stream and non-base view stream, scaling processingis not performed, and the left eye image data and right eye image datais output without change.

The L/R reconfiguring unit 263 reconfigures the left eye image data andright eye image data. That is to say, when at least one of the disparityangles in the same-side direction or intersecting direction exceed thepredetermined range where the health of the listener is not harmed, theL/R reconfiguring unit 263 reconfigures the left eye image data andright eye image data so as to be contained within the predeterminedrange, based on the above-described reconfiguring instructions from thedisparity information control unit 262.

The coded buffer 231 temporarily stores a subtitle stream extracted withthe demultiplexer 214. The subtitle decoder 232 performs oppositeprocessing from the subtitle encoder 117 of the transmission datagenerated unit 110 (see FIG. 8) described above. That is to say, thesubtitle decoder 232 performs decoding processing for the subtitlestream stored in the coded buffer 231, and obtains the subtitle data.

Bitmap data for the subtitle, the display position information “Subtitlerendering position (x2, y2)”, and the disparity information of thesubtitle “Subtitle disparity” are included in this subtitle data. Thepixel buffer 233 temporarily stores the subtitle bitmap data and thesubtitle display position information “Subtitle rendering position (x2,y2)” obtained by the subtitle decoder 232. The subtitle disparityinformation buffer 234 temporarily stores the disparity information ofthe subtitle “Subtitle disparity” obtained by the subtitle decoder 232.

The subtitle display control unit 235 generates bitmap data for thesubtitle and bitmap data of the subtitle “Subtitle data” for left eyedisplay and for right eye display to which disparity has been appended,based on the display position information and disparity information ofthe subtitle. The set-top box (STB) graphics generating unit 219generates graphics such as OSD or application, or graphics data such asEPG. Graphics bitmap data “Graphics data” and graphics display positioninformation “Graphics rendering position (x1, y1)” are included in thisgraphics data.

The graphics buffer 221 temporarily stores the graphics bitmap data“Graphics data” generated by the set-top box graphics generating unit219. The overlaying unit 217 overlays bitmap data for the subtitle“Subtitle data” for the left eye display and for the right eye display,which is generated by the subtitle display control unit 235, over theleft eye image data and right eye image data, respectively.

Also, the overlaying unit 217 overlays the graphics bitmap data“Graphics data” that is stored in the graphics buffer 221 over each ofthe left eye image data and right eye image data. In this event,disparity is appended to the graphics bitmap data “Graphics data” thatis overlaid over each of the left eye image data and right eye imagedata by a later-described depth control unit 220. Now, in the case thatthe graphics bitmap data “Graphics data” shares the same pixel as thebitmap data “Subtitle data” of the subtitle, the overlaying unit 217overwrites the graphics data over the subtitle data.

The depth control unit 220 appends disparity to the graphics bitmap data“Graphics data” overlaid onto each of the left eye image data and righteye image data. Therefore, the depth control unit 220 generates displayposition information “Rendering position” of the graphics for the lefteye display and for the right eye display, for each picture of the imagedata, and performs shifting control of the overlay position to the lefteye image data and right eye image data of the graphics bitmap data“Graphics data” stored in the graphics buffer 221.

The depth control unit 220 uses the information below to generatedisplay position information “Rendering position”, as illustrated inFIG. 31. That is to say, the depth control unit 220 uses the firstdisparity information (Min disparity) for each picture output from thedisparity information control unit 262. Also, the depth control unit 220uses the display position information “Subtitle rendering position (x2,y2)” of the subtitle stored in the pixel buffer 233.

Note that if we focus on changes to the depth for each partition regionin an active depth update, in the event that an object in a backgroundstereoscopic image changes spatial position, the change amount of depthinstantly becomes significant, and if this is used without modificationfor overlay on graphics, it goes without saying that the viewer willexperience discomfort regarding depth. Therefore, in order to suppresssuch discomfort, the depth control unit 220 performs processing(filtering processing) described below as to the first disparityinformation (Min disparity) of each partition region for each picture,that smoothes in the temporal direction, for example, and suppressessudden changes to be smooth changes. Now, the first disparityinformation is a received absolute value that has been converted to anegative value.

FIG. 32 illustrates an example of filtering processing. This example isan example in a case where the disparity information value at timing T0is A, the disparity information value at timing T1 is B, and change issudden. In this case, the disparity information value at timing T1 isnot B itself, but rather (A−α1). Similarly, this is (A−α1*2) at timingT2 and (A−α1*3) at timing T3. Here, α1=(A−B)/N. Thus, filteringprocessing is performed so that the disparity information difference of(A−B) changes gradually over time of N video frames, and finally thedisparity information value settles down to B.

Now, in the case that the (A−B) disparity information difference is Npixels, for example, changes are made over a period of N video frames ata change rate of one pixel for one frame. Also, in the case that the(A−B) disparity information difference is not an integer multiple of Npixels, a decimal value occurs in the change amount for each frame, buta so-called sub-pixel control can also be performed, where graphics tobe overlaid onto the image is subjected to interpolating,expanding/shrinking and so forth.

FIG. 33 illustrates another example of filtering processing. Thisexample is an example in a case where the disparity information value attiming T0 is A, the disparity information value at each of timings T1,T2, and T3 is B, C, D, and change is sudden. In this case, the disparityinformation value at timing T1 is not B itself, but rather (A−α1). Here,α1=(A−B)/N. In this case, filtering processing is performed so that thedisparity information difference of (A−B) changes gradually over time ofN video frames.

Also, the disparity information value at timing T2 is not C itself, butrather is (A−α1)−α2. Here, α2=((A−α1)−C)/P. In this case, filteringprocessing is performed so that the disparity information difference of((A−α1)−C) changes gradually over time of P video frames.

Further, the disparity information value at timing T2 is not D itself,but rather is ((A−α1)−α2)−α3). Here, α3=(((A−α1)−α2)−D)/Q. In this case,filtering processing is performed so that the disparity informationdifference of (((A−α1)−α2)−D) changes gradually over time of Q videoframes, and finally the disparity information value settles down to D.

Note that the processing for smoothing in the temporal direction(filtering processing) is not limited to the above-described example,and other methods may be used. The main point is that sudden changes todisparity information can be suppressed by this filtering processing.

Returning to FIG. 31, also, the depth control unit 220 uses disparityinformation “Subtitle disparity” of the subtitle stored in the subtitledisparity information buffer 234. Also, the depth control unit 220 usesgraphics display position information “Graphics rendering position (x1,y1)” that is generated by the set-top box graphics generating unit 219.Also, the depth control unit 220 uses identification information thatindicates whether or not disparity information is inserted into thevideo stream.

Also, the depth control unit 220 updates the first disparity information(Min disparity) for each picture output from the disparity informationcontrol unit 262, according to the overlaying of a subtitle or graphicsonto the image. In this case, the depth control unit 220 updates thedisparity information value of the partition region corresponding to thesubtitle display position and the graphics display position to a valueof the disparity information uses to append disparity to the subtitle orgraphics.

The flowcharts in FIG. 34 and FIG. 35 illustrate an example ofprocedures of the control processing of the depth control unit 220. Thedepth control unit 220 executes this control processing for each picture(frame) in which graphics display is performed. The depth control unit220 starts the control processing in step ST1. Subsequently, in stepST2, determination is made as to whether or not disparity informationfor graphics has been inserted into the video stream, based on theidentification information.

When disparity information for graphics has been inserted into the videostream, the depth control unit 220 moves to the processing in step ST3.In this step ST3, all of the partition regions including coordinates forgraphics overlay display are inspected. Also, in step ST4, the depthcontrol unit 220 compares the disparity information of the partitionregion to be used, selects an optimal value, for example a minimumvalue, for the value of the graphics disparity information(graphics_disparity).

Next, the depth control unit 220 moves to the processing in step ST5.When disparity information has not been inserted into the video streamin the step ST2 described above, the depth control unit 220 moves to theprocessing in step ST5. In this step ST5, the depth control unit 220determines whether or not there is a subtitle stream having disparityinformation.

When there is a subtitle stream having disparity information, in stepST6 the depth control unit 220 compares the value of the disparityinformation for subtitles (subtitle_disparity) and the disparityinformation value for graphics (graphics_disparity). Note that whendisparity information for graphics (disparity) has not been insertedinto the video stream, the disparity information value for graphics(graphics_disparity) will be “0”, for example.

Next, the depth control unit 220 determines in step ST7 whether or notthe condition of “subtitle_disparity>(graphics_disparity) is satisfied.When this condition is satisfied, in step ST8 the depth control unit 220uses the same value as the disparity information value for graphics(graphics_disparity) as to the graphics bitmap data “Graphics data”stored in the graphics buffer 221, obtains graphics bitmap data for lefteye display and for right eye display in which the display positionshave been shifted, and overlays these to the left eye image data andright eye image data, respectively.

Next, in step ST9, the depth control unit 220 updates the value of thedisparity information of the partition region corresponding to thescreen position on which the subtitle or graphics are overlaid. Afterthe processing in step ST9, in step ST10 the depth control unit 220 endsthe control processing.

On the other hand, when conditions are not satisfied in step ST7, instep ST10 the depth control unit 220 uses a value smaller than subtitledisparity information as to the graphics bitmap data “Graphics data”stored in the graphics buffer 221, obtains graphics bitmap data for lefteye display and for right eye display in which the display positionshave been shifted, and overlays these to the left eye image data andright eye image data, respectively. After the processing of step ST11,the depth control unit 220 goes through the processing of step ST9 andends the control processing in step ST10.

Also, when there is no subtitle stream having disparity information instep ST5, the depth control unit 220 moves to the processing in stepST12. In this step ST12, the depth control unit 220 performs graphicsdepth control by using the value for graphics disparity information(graphics_disparity) obtained in step ST4 or the value of disparityinformation calculated by the set-top box 200.

That is to say, the depth control unit 220 uses the value for graphicsdisparity information (graphics_disparity) or the value of thecalculated disparity information as to the graphics bitmap data“Graphics data” stored in the graphics buffer 221, obtains graphicsbitmap data for left eye display and for right eye display in which thedisplay positions have been shifted, and overlays these to the left eyeimage data and right eye image data, respectively. After the processingin step ST12, in step ST10 the depth control unit 220 ends the controlprocessing via the processing in step ST9.

FIG. 36 illustrates a graphics depth control example with the set-topbox 200. In this example, the graphics (STB graphics) have disparityappended to the graphics for the left eye display and graphics for theright eye display, based on the disparity information of the minimumvalue from the disparity information of eight partition regions on theright side (Partition 2, 3, 6, 7, 10, 11, 14, 15). Consequently, thegraphics are displayed nearer than the image (video) object of theseeight partition regions.

FIG. 37 also illustrates a graphics depth control example with theset-top box 200. In this example, the graphics (STB graphics) havedisparity appended to the graphics for the left eye display and graphicsfor the right eye display, based on the disparity information of theminimum value from the disparity information of eight partition regionson the right side (Partition 2, 3, 6, 7, 10, 11, 14, 15) and furtherbased on the disparity information of the subtitle.

Consequently, the graphics are displayed nearer than the image (video)object of these eight partition regions, and further, are displayednearer than the subtitle. Note that in this case, the subtitle is alsodisplayed nearer than the image (video) object of the four partitionregions (Partition 8, 9, 10, 11) corresponding to the subtitle displaypositions.

Note that updating processing of the disparity information in the caseof the depth control example in FIG. 37 is performed as follows. That isto say, first, the disparity information values of four partitionregions (Partition 8, 9, 10, 11) corresponding to the subtitle displaypositions are updated with the disparity information values(subtitle_disparity) used to append disparity to the subtitles.Subsequently, the disparity information values of eight partitionregions (Partition 2, 3, 6, 7, 10, 11, 14, 15) are updated with thedisparity information values (graphics_disparity) used to appenddisparity to the graphics.

Returning to FIG. 28, the coded buffer 241 temporarily stores the audiostream extracted by the demultiplexer 212. The audio decoder 242performs processing that is opposite to the audio encoder 119 (see FIG.8) of the transmission data generating unit 110 described above. That isto say, the audio decoder 242 performs decoding processing on the audiostream stored in the coded buffer 241, and obtains decoded audio data.The audio buffer 243 temporarily stores the audio data obtained by theaudio decoder 242. The channel mixing unit 244 generates and outputsaudio data for each channel to realize 5.1 ch surround or the like, forexample, as to audio data stored in the audio buffer 243.

Note that reading out of information (data) from the decoded buffer 215,disparity information buffer 218, pixel buffer 233, subtitle disparityinformation buffer 234, and audio buffer 243 is performed based on PTS,and transfer synchronization is performed.

The HDMI transmitting unit 251 transmits left eye image data and righteye image data obtained by subtitle and graphics overlay processinghaving been performed at the overlaying unit 217, and audio data of eachchannel obtained at the channel mixing unit 244, to an HDMI sink device,the television receiving device 300 according to the present embodiment,by communication conforming to HDMI. Now, the left eye image dataobtained with the overlaying unit 217 is left eye image data on whichsubtitles and STB graphics for the left eye display are overlaid. Also,the right eye image data obtained with the overlaying unit 217 is righteye image data on which subtitles and STB graphics for the right eyedisplay are overlaid.

Also, this HDMI transmitting unit 251 transmits the first disparityinformation (Min disparity) of each picture updated by the depth controlunit 220 and the second disparity information (Min disparity) of eachpicture output from the disparity information control unit 262 to thetelevision receiving device 300 by way of an HDMI interface. Accordingto the present embodiment, this first disparity information and seconddisparity information are inserted into an image data blanking period oran active space region, and transmitted. Details of this HDMItransmitting unit 251 will be described later.

Operations of the set-top box 200 illustrated in FIG. 28 will bedescribed briefly. The transport stream TS received by a digital tuneror the like is temporarily stored in the container buffer 211. A videostream, subtitle stream, and audio stream are included in this transportstream TS. As a video stream, one or two video streams obtained by theleft eye image data and right eye image data being encoded are included.

With the demultiplexer 212, streams for each of video, subtitles, andaudio are extracted from the transport stream TS that has beentemporarily stored in the container buffer 211. Also, with thedemultiplexer 212, identification information indicating whether or notdisparity information is inserted in the video stream (flag informationof “graphics_depth_info_not_existed_flag[0]”) is extracted from thistransport stream TS, and is sent to an unshown control unit (CPU).

The video stream extracted by the demultiplexer 212 is supplied to thecoded buffer 213 and temporarily stored. Also, decoding processing ofthe video stream stored in the coded buffer 213 is performed in thevideo decoder 214, and left eye image data and right eye image data isobtained. The left eye image data and right eye image data aretemporarily stored in the decoded buffer 215.

Also, disparity information (first disparity information, seconddisparity information) for each picture of image data inserted into thevideo stream is obtained by the video decoder 214. This disparityinformation is temporarily stored in the disparity information buffer218. In the disparity information control unit 262, based on the firstdisparity information and second disparity information for each picturestored in the disparity information buffer 218, checks are performed asto whether or not the disparity angle corresponding to the nearestobject playing position (disparity angle in the intersecting direction)and the disparity angle corresponding to the farthest object playingposition (disparity angle in the same-side direction) are containedwithin a range d (see FIG. 29) that does not harm the health of theviewer.

Now, with this disparity information control unit 262, in the case thatthe disparity angles are not contained within a predetermined range,instructions to reconfigure the left eye image data and right eye imagedata are given to the L/R reconfiguring unit 263 so that the disparityangles are contained within the predetermined range. Also, in this case,with the disparity information control unit 262, the first disparityinformation and/or second disparity information are corrected and outputto match the reconfigured left eye image data and right eye image data.Note that with the disparity information control unit 262, in the casethat the disparity angles are contained within a predetermined range d.The L/R reconfiguring unit 263 is not instructed to reconfigure the lefteye image data and right eye image data, and also, the first disparityinformation and second disparity information are output without beingcorrected.

With the scaler 216, scaling processing in the horizontal direction andthe vertical direction as to the left eye image data and right eye imagedata output from the decoded buffer 215 is performed, as necessary. Lefteye image data and right eye image data of full-HD that is 1920*1080,for example, is obtained from this scaler 216. This left eye image dataand right eye image data is supplied to the overlaying unit 217, via theL/R reconfiguring unit 263.

Reconfiguring of left eye image data and right eye image data isperformed, as needed, with the L/R reconfiguring unit 263. That is tosay, with the L/R reconfiguring unit 263, when one of the disparityangles in the same-side direction or the intersecting direction is notcontained within the range d (see FIG. 29) which does not harm thehealth of the viewer, reconfiguring the left eye image data and righteye image data is performed so that the disparity angles thereof arecontained within the predetermined range, based on reconfiguringinstructions from the disparity information control unit 262.

Also, the subtitle stream extracted by the demultiplexer 212 is suppliedto the coded buffer 231 and temporarily stored. With the subtitledecoder 232, decoding processing of the subtitle stream stored in thecoded buffer 231 is performed, and subtitle data is obtained. Bitmapdata of the subtitle, display position information “Subtitle renderingposition (x2, y2)” of this subtitle, and disparity information of thesubtitle “Subtitle disparity” are included in this subtitle data.

The bitmap data of the subtitle and the display position information“Subtitle rendering position (x2, y2)” of this subtitle obtained by thesubtitle decoder 232 are temporarily stored in the pixel buffer 233.Also, the subtitle disparity information “Subtitle disparity” obtainedat the subtitle decoder 232 is temporarily stored in the subtitledisparity information buffer 234.

Subtitle bitmap data “Subtitle data” for left eye display and for righteye display to which disparity has been appended is generated by thesubtitle display control unit 235, based on the subtitle bitmap data andthe display position information and disparity information of thissubtitle. The subtitle bitmap data “Subtitle data” data for left eyedisplay and for right eye display thus generated is supplied to theoverlaying unit 217, and overlaid over the left eye image data and righteye image data, respectively.

OSD or applications, or graphics data such as EPG are generated in theset-top box (STB) graphics generating unit 219. Graphics bitmap data“Graphics data” and the display position information of these graphics“Graphics rendering position (x1, y1)” are included in this graphicsdata. Graphics data generated by the set-top box (STB) graphicsgenerating unit 219 is temporarily stored in the graphics buffer 221.

The graphics bitmap data “Graphics data” that is stored in the graphicsbuffer 221 is overlaid over the left eye image data and right eye imagedata at the overlaying unit 217. In this event, the of the firstdisparity information for each partition region of each picture of theimage data output from the disparity information control unit 262,disparity is appended to the graphics bitmap data “Graphics data” thatis overlaid onto each of the left eye image data and right eye imagedata, based on disparity information corresponding to the graphicsdisplay position, by the depth control unit 220.

In this case, in the case that the graphics bitmap data “Graphics data”shares the same pixels as the subtitle bitmap data “Subtitle data”, thegraphics data is overwritten over the subtitle data by the overlayingunit 217. Also, in this case, as described above, the first disparityinformation is not used without change, but is smoothed in the temporaldirection and used by the depth control unit 220 in order to preventdiscomfort of depth perception of the graphics overlay.

The left eye image data, over which the subtitle and graphics for theleft eye display is overlaid, and the right eye image data, over whichthe subtitle and graphics for the right eye display is overlaid, areobtained from the overlaying unit 217. This left eye image data andright eye image data are supplied to the HDMI transmitting unit 251.

Also, the audio stream extracted by the demultiplexer 212 is supplied tothe coded buffer 241 and temporarily stored. With the audio decoder 242,decoding processing of the audio stream stored in the coded buffer 241is performed, and decoded audio data is obtained. This audio data issupplied to the channel mixing unit 244 via the audio buffer 243. Audiodata for each channel to realize a 5.1 ch sound or the like as to theaudio data, for example, is generated by the channel mixing unit 244.This audio data is supplied to the HDMI transmitting unit 251.

Also, with the depth control unit 220, the first disparity informationfor each partition region of each picture of the image data output fromthe disparity information control unit 262 is updated according tooverlaying of subtitles or graphics onto the image. In this case, thevalues of the disparity information of the partition regioncorresponding to the subtitle display position and graphics displayposition are updated to the values of the disparity information used toappend disparity to subtitles or graphics, for example. This updateddisparity information is supplied to the HDMI transmitting unit 251.Also, the second disparity information of each picture of the image dataoutput from the disparity information control unit 262 is also suppliedto the HDMI transmitting unit 251.

Left eye image data and right eye image data, audio data, and further,disparity information (first disparity information, second disparityinformation) of each picture of the image data, is transmitted by theHDMI transmitting unit 251, with communication that is HDMI-compliant,to the television receiving device 300. The details of the HDMItransmitting unit 251 will be described later.

The disparity information is inserted in a blanking period of the imagedata or an active video space, and transmitted. Specifically, an HDMIVendor Specific InfoFrame serving as an information packet that isdisposed in the blanking period of the image data, for example, may beused. Also, for example, a data packet disposed in a data island periodthat is newly defined, for example, may be used. Also, for example, anactive space region, which exists in an active video space, may be used.

[Description of Television Receiving Device]

Returning to FIG. 1, the television receiving device 300 receives lefteye image data and right eye image data, audio data, and further,disparity information (first disparity information, second disparityinformation) of each picture of the image data that is transmitted fromthe set-top box 200 via the HDMI cable 400.

In the event of performing overlay display of graphics (TV graphics)onto an image, for example, the television receiving device 300 uses theimage data and first disparity information and the graphics data toobtain the left eye image and right eye image data on which the graphicsare overlaid. In this case, the television receiving device 300 appendsdisparity corresponding to the display position of the graphics, foreach picture, to the graphics that are overlaid onto the left eye imageand right eye image, and obtains the data of the left eye image overwhich graphics are overlaid and data of the right eye image over whichgraphics are overlaid.

By appending disparity to the graphics as described above, the graphics(TV graphics) that are overlaid and displayed over a stereoscopic imagecan be displayed nearer than an object in the stereoscopic image at thedisplay position thereof. Thus, in the case of overlaying and displayinggraphics of an OSD or an application or EPT of program information orthe like, consistency in perspective as to each object within the imagecan be maintained.

Also, the television receiving device 300 can check whether or not thedisparity angle corresponding to the nearest object playing position(disparity angle in the intersecting direction) and the disparity anglecorresponding to the farthest object playing position (disparity anglein the same-side direction) are contained within a range d (see FIG. 47)that does not harm the health of the viewer, based on the firstdisparity information and the second disparity information, and if notcontained, can reconfigure the left eye image data and right eye imagedata.

[Configuration Example of Television Receiving Device]

FIG. 38 illustrates a configuration example of an HDMI input system ofthe television receiving device 300. Note that the checking system fordisparity angles is omitted. The television receiving device 300 has anHDMI receiving unit 311, scaler 312, overlaying unit 313, depth controlunit 314, graphics buffer 315, television (TV) graphics generating unit316, and audio processing unit 317.

The HDMI receiving unit 311 receives left eye image data and right eyeimage data that configures the stereoscopic image and audio data from anHDMI source device, which is the set-top box 200 according to thepresent embodiment, from communication that is HDMI-compliant. Also,this HDMI receiving unit 311 receives disparity information (firstdisparity information, second disparity information) for each picture ofthe image data from the set-top box 200 with an HDMI interface. Detailsof this HDMI receiving unit 311 will be described later.

The scaler 312 performs scaling processing as needed on the left eyeimage data and right eye image data received by the HDMI receiving unit311. For example, the scale 312 matches the sizes of the left eye imagedata and right eye image data to a display size. The television (TV)graphics generating unit 316 generates an OSD or application or graphicsdata such as an EPG. Graphics bitmap data “Graphics data” and displayposition information “Graphics rendering position (x1, y1)” of thegraphics thereof are included in this graphics data.

The graphics buffer 315 temporarily stores the graphics bitmap data“Graphics data” generated by the television graphics generating unit316. The overlaying unit 313 overlays the graphics bitmap data “Graphicsdata” stored in the graphics buffer 315 to each of the left eye imagedata and right eye image data. In this event, disparity is appended tothe graphics bitmap data “Graphics data” that is overlaid onto each ofthe left eye image data and right eye image data, by the later-describeddepth control unit 314.

The depth control unit 314 appends disparity to the graphics bitmap data“Graphics data” that is overlaid onto each of the left eye image dataand right eye image data. Therefore, the depth control unit 314generates display position information “Rendering position” of thegraphics for the left eye display and for the right eye display, foreach picture of the image data, and performs shifting control of theoverlay position to the left eye image data and right eye image data ofthe graphics bitmap data “Graphics data” stored in the graphics buffer315.

The depth control unit 314 generates a display position information“Rendering position”, using the information below, as illustrated inFIG. 39. That is to say, the depth control unit 314 uses the firstdisparity information (Min disparity) of each partition region for eachpicture of the image data received by the HDMI receiving unit 311. Also,the depth control unit 314 uses the graphics display positioninformation “Graphics rendering position (x1, y1)” generated by thetelevision graphics generating unit 316. Also, the depth control unit314 uses receiving information that indicate whether or not thedisparity information is received by the HDMI receiving unit 311.

The flowchart in FIG. 40 illustrates an example of procedures of controlprocessing of the depth control unit 314. The depth control unit 314executes this control processing with each picture (frame) that performsgraphics display. The depth control unit 314 starts the controlprocessing in step ST21. Subsequently, in step ST22, determination ismade as to whether or not disparity information for graphics has beenreceived by the HDMI receiving unit 311. Note that the when the “PRTY”identification information of the packet of the HDMI Vendor SpecificInfoFrame, which is to be described later, indicates the existence ofdisparity information as information to be referenced, the HDMIreceiving unit 311 extracts the disparity information from the packet,and provides for use. In this case, the receiving information is “thereis reception”.

When there is reception of disparity information, the depth control unit314 moves to the processing in step ST23. In step ST23, all of thepartition regions wherein coordinates for overlaying and displayinggraphics are included are inspected. Also, in step ST24, the depthcontrol unit 314 compares the first disparity information (Mindisparity) of the partition regions to be used, selects an optimalvalue, e.g. a minimum value, and sets this as the value of the graphicsdisparity information (graphics_disparity).

Next, in step ST25, the depth control unit 314 uses an equivalent valueas to the disparity information value for graphics (graphics_disparity)as to the graphics bitmap data “Graphics data” stored in the graphicsbuffer 315, obtains graphics bitmap data for left eye display and forright eye display in which the display positions have been shifted, andoverlays these to the left eye image data and right eye image data,respectively. After the processing in step ST25, in step ST26 the depthcontrol unit 314 ends the control processing.

Also, when there is no reception of disparity information in step ST22,in step ST27 the depth control unit 314 uses the value of the disparityinformation calculated by the television receiving device 300 as to thegraphics bitmap data “Graphics data” stored in the graphics buffer 315,obtains graphics bitmap data for left eye display and for right eyedisplay in which the display positions have been shifted, and overlaysthese to the left eye image data and right eye image data, respectively.After the processing in step ST27, in step ST26 the depth control unit314 ends the control processing.

FIG. 41 illustrates a depth control example of graphics in thetelevision receiving device 300. In this example, disparity is appendedto the graphics for left eye display and the graphics for right eyedisplay, based on the disparity information of the minimum value of thefirst disparity information of four partition regions on the right side(Partition 10, 11, 14, 15). As a result, the TV graphics are displayednearer than the image (video) object in these four partition regions.Note that in this case, the subtitles and further, the STB graphics, arealready overlaid onto the image (video) by the set-top box 200.

Operations of the television receiving device 300 illustrated in FIG. 38will be briefly described. With the HDMI receiving unit 311, left eyeimage data and right eye image data, audio data, and further, disparityinformation (first disparity information, second disparity information)for each picture of the image data is received from the set-top box 200by communication that is HDMI-compliant.

Upon scaling processing having been performed as needed by the scaler312, the left eye image data and right eye image data received by theHDMI receiving unit 311 is supplied to the overlaying unit 313. With thetelevision TV) graphics generating unit 316, an OSD or application orgraphics data such as EPG is generated. Graphics bitmap data “Graphicsdata” and the display position information of these graphics “Graphicsrendering position (x1, y1)” are included in this graphics data. Thegraphics data generated by the television graphics generating unit 315is temporarily stored in the graphics buffer 315.

The graphics bitmap data “Graphics data” stored in the graphics buffer315 is overlaid onto the left eye image data and right eye image data bythe overlaying unit 313. In this event, disparity is appended to thegraphics bitmap data “Graphics data”, which is overlaid onto each of theleft eye image data and right eye image data, by the depth control unit314, based on the first disparity information (Min disparity)corresponding to the graphics display position.

The first disparity information of each partition region for eachpicture of the image data, and the graphics display position information“Graphics rendering position (x1, y1)” generated by the televisiongraphics generating unit 316, which are received by the HDMI receivingunit 311, are used by the depth control unit 314 for the controlthereof.

Left eye image data onto which TV graphics for left eye display isoverlaid is obtained, and right eye image data onto which TV graphicsfor right eye display is obtained, from the overlaying unit 313. Thisimage data is transmitted to the processing unit for stereoscopic imagedisplay, and stereoscopic image display is performed.

Also, the audio data from each channel received by the HDMI receivingunit 311 is supplied to a speaker via the audio processing unit 317which performs adjustments to sound quality and volume, and audio outputthat matches the stereoscopic image display is performed.

[Configuration Example of HDMI Transmitting Unit and HDMI ReceivingUnit]

FIG. 42 illustrates a configuration example of the HDMI transmittingunit 251 of the set-top box 200 and the HDMI receiving unit 311 of thetelevision receiving device 300 in the image transmitting/receivingsystem 10 in FIG. 1.

The HDMI transmitting unit 251 transmits a differential signalcorresponding to image data of a non-compressed image for one screen, inone direction to the HDMI receiving unit 311, with multiple channels, invalid image segments (hereinafter, also called active video spaces).Now, the valid image segment is a segment that has removed horizontalblanking interval and vertical blanking interval from a segment of onevertical synchronized signal to the next vertical synchronized signal.Also, the HDMI transmitting unit 251 transmits in one direction to theHDMI receiving unit 311, a differential signal that corresponds to atleast audio data and control data, and other auxiliary data or the likethat is associated with the image, with multiple channels, in ahorizontal blanking interval or vertical blanking interval.

A transmission channel of the HDMI system made up of the HDMItransmitting unit 251 and HDMI receiving unit 311 may be the followingtransmission channel. That is to say, there are three TMDS channels, #0through #2, which are transmission channels that synchronize the imagedata and audio data to a pixel clock, and transmit serially in onedirection from the HDMI transmitting unit 251 to the HDMI receiving unit311. Also, there is a TMDS clock channel, which is a transmissionchannel to transmit pixel clocks.

The HMDI transmitting unit 251 has an HDMI transmitter 81. Thetransmitter 81 converts the image data of a non-compressed image to acorresponding differential signal, and through the three TMDS channels#0, #1, and #2 which are multiple channels, transmits serially in onedirection to the HDMI receiving unit 311 which is connected via the HDMIcable 400.

Also, the transmitter 81 converts the audio data associated to anon-compressed image, and further, other auxiliary data or the like suchas necessary control data, to a corresponding differential signal, andthrough the three TMDS channels #0, #1, and #2, transmits serially inone direction to the HDMI receiving unit 311.

Further, the transmitter 81 transmits a pixel clock that is synchronizedto the image data transmitted through the three TMDS channels #0, #1,and #2 to the HDMI receiving unit 311 connected via the HDMI cable 400,through a TMDS clock channel. Now, 10 bits of image data are transmittedin the time of one clock of a pixel clock, through one TMDS channel #i(i=0, 1, 2).

The HDMI receiving unit 311 receives the differential signalcorresponding to the pixel data, which is transmitted from the HDMItransmitting unit 251 in one direction through multiple channels, in anactive video space. Also, this HDMI receiving unit 311 receivesdifferential signals corresponding to the audio data and control datatransmitted in one direction from the HDMI transmitting unit 251,through multiple channels, in a horizontal blanking interval or verticalblanking interval.

That is to say, the HDMI receiving unit 311 has an HDMI receiver 82.This HDMI receiver 82 receives a differential signal corresponding topixel data and differential signals corresponding to audio data andcontrol data, which are transmitted in one direction from the HDMItransmitting unit 251 through the TMDS channels #0, #1, and #2. In thiscase, the signals transmitted from the HDMI transmitting unit 251through the TMDS clock channel are synchronized to the pixel clock, andreceived.

The transmission channel of the HDMI system may be a transmissionchannel called DDC (Display Data Channel) 83 or CEC line 84, besides theabove-described TMDS channels #0 through #2 and the TMDS clock channel.The DDC 83 is made up of two unshown signal lines included in the HDMIcable 400. The DDC 83 is used for the HDMI transmitting unit 251 to readout E-EDID (Enhanced Extended Display Identification Data) from the HDMIreceiving unit 311.

That is to say, the HDMI receiving unit 311 has an EDID ROM (Read OnlyMemory) 85, which stores the E-EDID which is capability informationrelating to its own capability (Configuration/capability), besides theHDMI receiver 81. The HDMI transmitting unit 251 reads out the E-EDIDfrom the HDMI receiving unit 311 that is connected via the HDMI cable400, via the DDC 83, according to a request from an unshown control unit(CPU), for example.

The HDMI transmitting unit 251 transmits the read-out E-EDID to thecontrol unit (CPU). Based on this E-EDID, the control unit (CPU) canrecognize the settings of the capabilities of the HDMI receiving unit311. For example, the control unit (CPU) recognizes whether or not thetelevision receiving device 300 having the HDMI receiving unit 311 canhandle stereoscopic image data, and if capable, whether any TMDStransmission data configuration can be handled.

The CEC line 84 is made up of one unshown signal line included in theHDMI cable 400, and is used to perform bi-directional communication ofcontrol data between the HDMI transmitting unit 251 and HDMI receivingunit 311. This CEC line 84 makes up a control data line.

Also, a line (HPD line) 86 connected to a pin called an HPD (Hot PlugDetect) is included in the HDMI cable 400. The source device can usethis line 86 and detect connections of the sink device. Note that thisHPD line 86 is also used as an HEAC-line which makes up a bi-directionalcommunication path. Also, a line (power line) 87 that is use to supplypower from the source device to the sink device is included in the HDMIcable 400. Further, a utility line 88 is included in the HMDI cable 400.This utility line 88 is also used as an HEAC+ line which makes up abi-directional communication path.

FIG. 43 illustrates a configuration example of TMDS transmission data.This FIG. 43 illustrates a segment of various types of transmission datain the case that image data having horizontal×vertical dimensions of1920 pixels×1080 lines is transmitted through the TMDS channels #0, #1,and #2.

Three types of segments exits in a video field in which transmissiondata is transmitted through the three TMDS channels #0, #1, and #2 ofHDMI, according to the type of transmission data. These three types ofsegments are a video data period, data island period, and controlperiod.

Now, the video field segment is a segment from the active edge of acertain vertical synchronizing signal to the active edge of the nextvertical synchronizing signal. This video field segment can be dividedin a horizontal blanking period, vertical blanking period, and activevideo space (Active Video). This active video space is a segment whichhas removed the horizontal blanking period and vertical blanking periodfrom the video field segment.

The video data period is allocated to an active video space. In thisvideo data period, data of active pixels for 1920 pixels 1080 linesmaking up one screen of non-compressed image data is transmitted.

The data island period and control period are allocated to thehorizontal blanking period and vertical blanking period. In the dataisland period and control period, auxiliary data is transmitted. That isto say, the data island period is allocated to a portion of thehorizontal blanking period and vertical blanking period. With this dataisland period, of the auxiliary data, data that is not related tocontrol, for example packets of audio data or the like, is transmitted.

The control period is allocated to other portions than the horizontalblanking period and vertical blanking period. In this control period, ofthe auxiliary data, data that is related to control, for examplevertical synchronizing signals and horizontal synchronizing signals,control packets, and the like are transmitted.

[Disparity Information Transmitting/Receiving Method with HDMI]

A method to transmit/receive disparity information of each partitionregion for each picture of the image data with an HDMI interface will bedescribed.

“(1) Example of Using HDMI Vendor Specific InfoFrame”

The transmission of disparity information in each partition region foreach picture of the image data, using HDMI Vendor Specific InfoFrame(VS_Info) will be described.

According to this method, in the VS_Info, “HDMI_Video_Format=“010”” and“3D_Meta_present=1”, whereby “Vendor Specific InfoFrame extension” isspecified. In this case, “3D_Metadata_type” is defined as an unused“001”, for example, and disparity information of each partition regionis specified.

FIG. 44 illustrates a VS_Info packet configuration example. This VS_Infois defined in CEA-861-D, so detailed information will be omitted. FIG.45 illustrates content of the primary information in the packetconfiguration ample illustrated in FIG. 44.

3-bit information “HDMI_Video_Format” indicating the type of image datais disposed from the seventh bit to the fifth bit of the fourth byte(PB4). In the case that the image data is 3D image data, this 3-bitinformation is “010”. Also, in the case that the image data is 3D imagedata, 4-bit information “3D_Structure” indicating a TMDS transmissiondata configuration is disposed from the seventh bit to the fourth bit ofthe fifth byte (PB5). For example, in the case of a frame packingmethod, this 4-bit information is “0000”.

Also, in the case that “3D_Meta_present” is disposed in the third bit ofthe fifth byte (PB5) and Vendor Specific InfoFrame extension isspecified, this 1-bit is “1”. Also, “3D_Metadata_type” is disposed fromthe seventh bit to the fifth bit of the seventh byte (PB7). In the caseof specifying disparity information of each partition region, this 3-bitinformation is an unused “001”, for example.

Also, “3D_Metadata_length” is disposed from the fourth byte to the 0'thbyte of the seventh byte (PB7). This 5-bit information indicates thesize of the disparity information of each partition region.

Also, 1-bit identification information of “PRTY” is disposed in the 0thbit of the sixth byte (PB6). This identification information isinformation that the HDMI sink side should reference, and here indicateswhether or not the disparity information is included in this VS_Info.“1” indicates that information that the HDMI sink should reference isalways included. “0” indicates that information that the HDMI sinkshould reference is not necessarily included.

This 1-bit identification information of “PRTY” is disposed, whereby theHDMI sink, which is the television receiving device 300 according to thepresent embodiment, can determine whether or not the information thatshould be referenced is included in the VS_Info, even without inspecting“3D_Metadata_type” and below. Accordingly, at the HDMI sink, extractingprocessing of information to be referenced from the VS_Info can beperformed without waste due to this identification information, andprocessing load can be reduced.

Also, “partition_type” is disposed from the seventh bit to the fifth bitof the eighth byte (PB8). This 3-bit information indicates the partitiontype of the display screen for the subject picture. “000” indicatespartition type “type000” as indicated in FIG. 19(a). “001” indicatespartition type “type001” as indicated in FIG. 19(b). “010” indicatespartition type “type010” as indicated in FIG. 19(c). “011” indicatespartition type “type011” as indicated in FIG. 19(d). “100” indicatespartition type “type100” as indicated in FIG. 19(e). “101” indicatespartition type “type101” as indicated in FIG. 19(f).

Also, 1-bit identification information of “d_picture” is disposed in thefourth bit of the eighth byte (PB8). This identification informationindicates either single picture or double picture. “0” indicates asingle picture, i.e., that the mode is to transmit the amount of onepicture as disparity information of each partition region. “1” indicatesa double picture, i.e., that the mode is to transmit the amount of twopictures as disparity information of each partition region.

Also, “partition_count” is disposed from the third bit to the 0'th bitin the eighth byte (PB8). This 4-bit information indicates the totalnumber of partition regions, and is a value that depends on theabove-described “partition_type”. For example, “0000” indicates 1,“0011” indicates 4, “0111” indicates 8, “1000” indicates 9, “1100”indicates 13, and “1111” indicates 16.

Also, at the eighth+1 byte (PB8+1) and thereafter, disparity information(first disparity information, second disparity information) for onepicture or two pictures is sequentially disposed. The 8-bit informationof “Max_disparity_in_picture” indicates the second disparity informationof the entire picture display screen (entire picture), i.e., the maximumdisparity information (disparity value) of the entire picture. The 8-bitinformation of “Min_disparity_in_picture” indicates the first disparityinformation of each partition region, i.e., the minimum disparityinformation (disparity value) of each partition region.

FIG. 46 illustrates a VS_Info configuration example in the case of“d_picture=0”, that the mode is for single picture,“partition_type=010”, and the partition region is “16”. In this case,disparity information for each partition region for one picture isdisposed at the eighth+1 byte (PB8+1) and thereafter.

As described above, in the case that disparity information is insertedinto the video stream in picture increments, the set-top box 200 obtainsdisparity information for one picture at the timing for each picture ofthe image data (see FIG. 25). Also, as described above, in the case thatdisparity information is inserted into the video stream in GOPincrements, the set-top box 200 obtains the disparity information foreach picture within the GOP (disparity information set) together, andthe head timing of the GOP of the image data (see FIG. 26).

In either case, the set-top box 200 can optionally select either mode ofsingle picture or double picture, based on negotiation using the CECline 84 between the television receiving device 300, or from settingswith the EDIDROM 85, for example. In this case, the set-top box 200 canselect the mode according to the transmission band for transmittingdisparity information for each picture, or processing capability of theset-top box 200 and television receiving device 300, wherebytransmitting disparity information to the television receiving device300 can be favorably performed.

With the television receiving device 300, disparity information of allof the pictures can be accurately received, regardless of which mode oftransmission is used, based on the mode identification information ofthe “d_picture” disposed in the VS_Info and the identificationinformation of whether or not the above-described “PRTY” referenceinformation exists.

FIG. 47 schematically illustrates a case wherein the set-top box 200obtains the disparity information for one picture at the timing of eachpicture of the image data, and sequentially transmits to the televisionreceiving device 300 the disparity information for each pictureaccording to single picture mode. Also, FIG. 48 schematicallyillustrates a case wherein the set-top box 200 obtains the disparityinformation for one picture at the timing of each picture of the imagedata, and sequentially transmits to the television receiving device 300the disparity information for each picture according to double picturemode.

Also, FIG. 49 schematically illustrates a case wherein the set-top box200 obtains the disparity information for each picture within the GOPtogether at the head timing of the GOP of the image data, andsequentially transmits the disparity information for each pictureaccording to the single picture mode to the television receiving device300. Further, FIG. 50 schematically illustrates a case wherein theset-top box 200 obtains the disparity information for each picturewithin the GOP together at the head timing of the GOP of the image data,and sequentially transmits the disparity information for each pictureaccording to the double picture mode to the television receiving device300.

Note that in the description above, the set-top box 200 is described asbeing able to optionally select the mode of single picture or doublepicture. However, for example, when obtaining the disparity informationof each picture within the GOP at the head timing of the GOP of theimage data, transmission may be in single picture mode. In this case,the disparity information of each picture within the GOP is assigned toindividual pictures, and the disparity information for each individualpicture is sequentially transmitted in increments of pictures (see FIG.49). In this case, even in the case where the transmission band fortransmitting disparity information for each picture is small, thedisparity information of each picture can be favorably transmitted tothe television receiving device 300.

On the other hand, in the case that the set-top box 200 can onlytransmit VS_Info at the rate of once for every two video frames, or thetelevision receiving device 300 can only receive VS_Info at the rate ofonce for every two video frames, transmitting the disparity informationfor two video frames consecutively, with one VS_Info, may be conceived,as in FIG. 48.

Note that in double picture mode, the amount of transmission data can bereduced by setting the disparity information for the first picture orsecond picture as the differential data between the disparityinformation of one picture prior.

FIG. 51 and FIG. 52 illustrate another packet configuration example ofthe VS_Info (HDMI Vendor Specific InfoFrame). While detaileddescriptions will be omitted, the 0'th byte (PB0) through the sixth byte(PB6) are similar to the packet configuration example illustrated inFIG. 44 described above. FIG. 45 illustrates the content of the primaryinformation in the packet configuration example illustrated in FIG. 51and FIG. 52.

“3D_Metadata_type” is disposed from the seventh bit to the fifth bit ofthe seventh byte (PB7). IN the case of specifying disparity informationof each partition region, this 3-bit information may be an unused “001”,for example.

Also, “3D_Metadata_length” is disposed from the fourth byte to the 0'thbyte of the seventh byte (PB7). This 5-bit information indicates thesize of disparity information of each partition region. The value ofthis “3D_Metadata_length” takes the value of 0x00 to 0x16. For example,“00011” is expressed as 3 (in decimal numbers), and “11010” is expressedas 26 (in decimal numbers).

1-bit identification information of “d_picture” is disposed at theseventh bit of the eighth byte (PB8). This identification informationindicates single picture or double picture. “0” indicates a singlepicture, that is, a mode to transmit one picture worth of disparityinformation of each partition region. “1” indicates a double picture,that is, a mode to transmit two pictures worth of disparity informationof each partition region.

1-bit identification information of “partition_enable” is disposed atthe fifth bit of the eighth byte (PB8). This identification informationindicates whether or not the picture in question has disparityinformation of each partition region. “1” indicates that partitionregions are specified in horizontal and vertical directions, and thateach has disparity information. “0” indicates that the entire screen hasone set of disparity information.

1-bit identification information of “Picture_reorder” is disposed in thesixth bit of the eighth byte (PB8). In the case of transmitting a doublepicture, whether, in the transmission of two pictures (N, N+1), N isfirst temporally and N+1 is later, or whether N+1 is first and N islater, is indicated. “1” indicates that the (N+1) picture is first andthe value of the disparity information is expressed in 8 bits, and thatthe N picture is later and the differential value from the disparityinformation of the (N−1) picture is expressed in 4 bits. “0” indicatesthat the N picture is first and the value of the disparity informationis expressed in 8 bits, and that the (N+1) picture is later and thedifferential value from the disparity information of the N picture isexpressed in 4 bits.

Also, “partition_count” is disposed from the third bit to the 0'th bitof the eighth byte (PB8). This 4-bit information indicates the totalnumber of partition regions. For example, “0000” indicates 1, “0011”indicates 4, “0111” indicates 8, “1000” indicates 9, “1100” indicates13, and “1111” indicates 16.

Also, at the eighth+1 byte (PB8+1) and thereafter, disparity information(first disparity information, second disparity information) for onepicture or two pictures is sequentially disposed. The 8-bit informationof “max_disparity_in_picture” indicates the second disparity informationof the entire picture display screen (entire picture), i.e., the maximumdisparity information (disparity value) of the entire picture. The 8-bitinformation of “Min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e., the minimumdisparity information (disparity value) of each partition region.

FIG. 51 illustrates a VS_Info configuration example in the case of“d_picture=1”, that the mode is for double picture, “picture_reorder=0”,and the N picture is temporally first and the N+1 picture is later. Notethat this example indicates a case where “partition_count=1111” and thepartition region is “16”.

In this case, the second disparity information for the entire picturedisplay screen of the N picture, i.e., “Max_disparity_in_picture” whichis the maximum disparity information (Disparity value) of the entirepicture, is disposed at the eighth+1 byte (PB8+1) and thereafter. Also,the first disparity information of each partition region of the Npicture, i.e., “Min_disparity_in_partition” which is the minimumdisparity information (Disparity value) of each partition region, isdisposed from the eighth+2 byte (PB8+2) to the eighth+16 byte (PB8+16).

Also, in this case, second disparity information of the entire picturedisplay screen of the N+1 picture, i.e., the“Differential_max_disparity_in_picture” which is differential data ofthe maximum disparity information (disparity value) of the entirepicture, is disposed from the third bit to the 0'th bit of the eighth+17byte (PB8+17). First disparity information of each partition region inthe N+1 picture, i.e., the “Differential_min_disparity_in_partition”which is differential data of the minimum disparity information(disparity value) of each partition region, is disposed from theeighth+18 byte (PB8+18) to the eighth+25 byte (PB8+25).

The configuration example of the VS_Info in FIG. 52 illustrates aVS_Info configuration example in the case of “d_picture=1”, that themode is for double picture, “picture_reorder=1”, and the N+1 picture istemporally first and the N picture is later. Note that this exampleindicates a case where “partition_count=1111” and the partition regionis “16”.

In this case, second disparity information for the entire picturedisplay screen of the N+1 picture, i.e., “Max_disparity_in_picture”which is the maximum disparity information (disparity value) of theentire picture, is disposed at the eighth+1 byte (PB8+1). Also, thefirst disparity information of each partition region of the N+1 picture,i.e., “Min_disparity_in_partition” which is the minimum disparityinformation (disparity value) of each partition region, is disposed fromthe eighth+2 byte (PB8+2) to the eighth+16 byte (PB8+16).

Also, in this case, second disparity information of the entire picturedisplay screen of the N picture, i.e., the“Differential_max_disparity_in_picture” which is differential data ofthe maximum disparity information (disparity value) of the entirepicture, is disposed from the third bit to the 0'th bit of the eighth+17byte (PB8+17). First disparity information of each partition region inthe N picture, i.e., the “Differential_min_disparity_in_partition” whichis differential data of the minimum disparity information (disparityvalue) of each partition region in the N picture, is disposed from theeighth+18 byte (PB8+18) to the eighth+25 byte (PB8+25).

[Order Determination of N Picture and N+1 Picture]

Now, the order determination of the N picture and N+1 picture, i.e., thedetermination of whether “picture_reorder” will be “0” or will be “1” isperformed as follows, for example. The order determination of the Npicture and N+1 picture is performed in a configuration as illustratedin FIG. 53, for example. The disparity information is supplied to aframe memory 281, and only one frame is delayed. With a subtractor 282,the differential data “D(N)−D(N+1)” between the N+1 picture disparityinformation D(N+1) and the N picture disparity information D(N) iscalculated, and the differential data herein is sent to an orderdetermining unit 283.

With the order determining unit 283, an absolute value of differencedata |D(N)−D(N+1)| and a threshold value Th are compared, and orderdetermination of the N picture and N+1 picture is performed. When|D(N)−D(N+1)|≦Th holds, the order determining unit 283 determines that“N picture is first, N+1 picture is later”, the “picture_reorder” of theVS_Info is set to “0”, and the disparity information of the N pictureand N+1 picture is disposed as illustrated in FIG. 51 described above.In this case, the disparity information of the N+1 picture is thedifferential data from the disparity information of the N picture.

FIG. 54 illustrates a time shift example of the disparity information(Disparity value) in the case that |D(N)−D(N+1)|≦Th holds. In thisexample, the disparity information changes greatly between the N−1picture and the N picture. However, change to the disparity informationbetween the N picture and the N+1 picture is small. Therefore,|D(N)−D(N+1)|≦Th is satisfied. In this case, the disparity informationof the N+1 picture is the differential data from the disparityinformation of the N picture, so the value thereof is relatively small.

On the other hand, when |D(N)−D(N+1)|>Th holds, the order determiningunit 283 determines that “N+1 picture is first, N picture is later”,sets the “picture_redorder” of the VS_Info to “1”, and disposes thedisparity information of the N+1 picture and N picture as illustrated inFIG. 52 described above. In this case, the disparity information of theN picture is the differential data from the disparity information of theN−1 picture.

FIG. 55 illustrates a time shift example of the disparity information(Disparity value) in the case that |D(N)−D(N+1)|>Th holds. In thisexample, change to the disparity information between the N−1 picture andthe N picture is small, but change to the disparity information betweenthe N picture and the N+1 picture is great. Therefore, |D(N)−D(N+1)|>This satisfied. In this case, the disparity information of the N pictureis the differential data from the disparity information of the N−1picture, so the value thereof is relatively small.

Now, regarding “Min_disparity_in_partition”, as the D(N+1), D(N), asshown in the Expressions (2), (3) below, the minimum value Min_disparity(N+1), Min_disparity (N) of the “Min_disparity_in_partition” for eachpartition region is used, respectively.

$\begin{matrix}\begin{matrix}{{D\left( {N + 1} \right)} = {{Min\_ disparity}\left( {N + 1} \right)}} \\{= {{Minimum}\mspace{14mu}\left( {{Min\_ disparity}{\_ partition}} \right.}} \\\left. \left( {{N + {1\_ 0}},{N + {1\_ 1}},{--{,{N + {1\_ 15}}}}} \right) \right)\end{matrix} & (2) \\\begin{matrix}{{D(N)} = {{Min\_ disparity}(N)}} \\{= {{Minimum}\mspace{14mu}\left( {{Min\_ disparity}{\_ partition}} \right.}} \\\left. \left( {{{N\_}0},{{N\_}1},{--{,{{N\_}15}}}} \right) \right)\end{matrix} & (3)\end{matrix}$

Note that instead of finding D(N) with the Expression (3) above,“Min_disparity_partition” of the N picture of the same partition regionas the “Min_disparity_partition” for Min_disparity (N+1) which is usedas D(N+1) in the above Expression (2) can be used.

On the other hand, regarding “Max_disparity_in_picture”,“Max_disparity_in_picture” of the N+1 picture is used as D(N+1), and“Max_disparity_in_picture” of the N picture is used as D(N). Note thatof the comparison determination results between |D(N)−D(N+1)| and Th, inthe case that the results relating to “Min_disparity_in_partition” andthe results relating to “Max_disparity_in_picture” do not match, theresults relating to “Min_disparity_in_partition” can be prioritized, andso forth.

Note that in the configuration example of the VS_Info in FIG. 52described above, the N+1 picture is temporally first and the N pictureis disposed later. Thus, the configuration from the eighth+1 byte(PB8+1) to the eighth+25 bytes (PB+25) has the same configurationexample as the VS_Info in FIG. 69 where the N picture is temporallyfirst and the N+1 picture is disposed later, whereby reading on thereceiving side can be stabilized. However, in the configuration exampleof the VS_Info illustrated in FIG. 52, the disposal position of thedisparity information of the N+1 picture and disparity information ofthe N picture may be reversed.

“(2) Example of Defining and Using New Data Packet”

A method to perform transmission of disparity information in eachpartition region for each picture of the image data, using a data packetthat has been newly defined in a data island period, will be described.

FIG. 56 illustrates a configuration example of a packet header of a 3DDisplaying Support Packet serving as the newly defined data packet.Hereinafter, this packet is called a “3DDS packet”. Content of theprimary information in the configuration example illustrated in FIG. 56is illustrated in FIG. 45.

The packet header of this 3DDS packet is a 3-byte configuration. Packettype information is disposed from the seventh bit to the 0'th bit of the0'th byte (HB0). Here, this is “0x0B” which indicates a 3DDS packet.

“3D_support_ID” is disposed in a 3-bit field from the seventh bit to thefifth bit of the first byte (HB1). This 3-bit information isidentification information to identify the data type (packet content)transmitted by 3DDS packets. “001” indicates disparity information ofeach partition region of the picture display screen.

“Payload length” is disposed from the fourth bit to the 0'th bit of thesecond byte (HB2). This 5-bit information indicates the size of packetcontents which is payload, continuing from this packet header, by bytelength.

FIG. 57 illustrates a configuration example of packet contents. Thisconfiguration example is an example of single picture mode. Contents ofthe primary information in the configuration example illustrated in thisFIG. 57 are illustrated in FIG. 45.

“3D_Metadata_type” is disposed from the seventh bit to the fifth bit ofthe 0'th byte (PB0). “001” illustrates that the disparity information isof each partition region of the picture display screen.“3D_Metadata_length” is disposed from the fourth bit to the 0'th bit ofthe 0'th byte (PB0). This 5-bit information illustrates the size ofdisparity information of each partition region.

“partition_type” is disposed from the sixth bit to the fourth bit of thefirst byte (PB1). This 3-bit information indicates the partition type ofthe display screen for the subject picture. “000” indicates partitiontype “type000” as indicated in FIG. 19(a). “001” indicates partitiontype “type001” as indicated in FIG. 19(b). “010” indicates partitiontype “type010” as indicated in FIG. 19(c). “011” indicates partitiontype “type011” as indicated in FIG. 19(d). “100” indicates partitiontype “type100” as indicated in FIG. 19(e). “101” indicates partitiontype “type101” as indicated in FIG. 19(f).

“partition_count” is disposed from the third bit to the 0'th bit of thefirst byte (PB1). This 4-bit information indicates the total number ofpartition regions (Partitions), and becomes a value that depends on theabove-described “partition_type”. For example, “0000” indicates 1,“0011” indicates 4, “0111” indicates 8, “1000” indicates 9, “1100”indicates 13, and “1111” indicates 16.

The disparity information (first disparity information, second disparityinformation) for one picture is disposed at the third byte (PB3) andthereafter. The 8-bit information of “Max_disparity_in_picture”indicates the second disparity information of the entire picture displayscreen (entire picture), i.e., the maximum disparity information(disparity value) of the entire picture. The 8-bit information of“Min_disparity_in_partition” indicates the first disparity informationof each partition region, i.e., the minimum disparity information(disparity value) for each partition region.

Now, “Min_disparity_in_partition” which is the first disparityinformation and “Max_disparity_in_picture” which is the second disparityinformation are absolute value data. “Min_disparity_in_partition” islimited to be nearer than the monitor position, whereby, on thereceiving side, the value (absolute value) of the transmitted disparityinformation can be converted to a negative value and used, even withoutindicating a sign bit. Similarly, “Min_disparity_in_picture” is limitedto be on the far side of the monitor position, whereby, on the receivingside, the value (absolute value) of the transmitted disparityinformation can be converted to a positive value and used, even withoutindicating a sign bit.

By the disparity information (first disparity information, seconddisparity information) thus becoming absolute value data, the dynamicrange of the disparity information expressed with 8 bits can be extendedto a range that is 0 to 255 pixels. Also, by adding the above-describedrestriction, a depth expression of up to −255 to +255 can be made.Therefore, even with a 4K*2K monitor, which is considered to havesuperhigh image quality resolution, depth control can be performed withcurrent transmission bands.

Note that the configuration example illustrated in FIG. 57 is an examplein the case of “3D_Metadata_length=11010”, “partition_type=101”,“partition_count=1111” (see FIG. 19(f)).

FIG. 58 illustrates another configuration example of the packet content.This configuration example is an example of double picture mode. FIG. 45illustrates content of the primary information in this configurationexample illustrated in FIG. 58.

“3D_Metadata_type” is disposed from the seventh bit through to fifth bitof the 0'th byte (PB0). “001” indicates disparity information of eachpartition region of the picture display screen. “3D_Metadata_length” isdisposed from the fourth bit through the 0'th bit of the 0'th byte(PB0). This 5-bit information indicates the size of the disparityinformation of each partition region.

1-bit identification information of “d_picture” is disposed at theseventh bit of the first byte (PB1). This identification informationindicates single picture or double picture. “0” indicates a singlepicture, i.e., a mode to transmit one picture as disparity informationof each partition region. “1” indicates a double picture, i.e., a modeto transmit two pictures as disparity information of each partitionregion.

1-bit identification information of “partition_enable” is disposed atthe fifth bit of the first byte (PB1). This identification informationindicates whether the subject picture has disparity information of eachpartition region. “1” indicates that partition regions are specified inthe horizontal and vertical directions, and that each has disparityinformation. “0” indicates that the entire screen has one set ofdisparity information.

1-bit identification information of “Picture_reorder” is disposed at thesixth bit of the first byte (PB1). In the case of transmitting a doublepicture, whether, in the transmission of two pictures (N, N+1), N isfirst temporally and N+1 is later, or whether N+1 is first and N islater, is indicated. “1” indicates that the (N+1) picture is first andthe value of the disparity information is expressed in 8 bits, and thatthe N picture is later and the differential value from the disparityinformation of the (N−1) picture is expressed in 4 bits. “0” indicatesthat the N picture is first and the value of the disparity informationis expressed in 8 bits, and that the (N+1) picture is later and thedifferential value from the disparity information of the N picture isexpressed in 4 bits.

Also, “partition_count” is disposed from the third bit to the 0'th bitof the first byte (PB1). This 4-bit information indicates the totalnumber of partition regions. For example, “0000” indicates 1, “0011”indicates 4, “0111” indicates 8, “1000” indicates 9, “1100” indicates13, and “1111” indicates 16.

Also, at the second byte (PB2) and thereafter, disparity information(first disparity information, second disparity information) for onepicture or two pictures is sequentially disposed. The 8-bit informationof “Max_disparity_in_picture” indicates the second disparity informationof the entire picture display screen (entire picture), i.e., the maximumdisparity information (disparity value) of the entire picture. The 8-bitinformation of “Min_disparity_in_partition” indicates the firstdisparity information of each partition region, i.e., the minimumdisparity information (disparity value) of each partition region.

Now, “Min_disparity_in_partition” which is the first disparityinformation and “Max_disparity_in_picture” which is the second disparityinformation are set as absolute value data, similar to the configurationexample in FIG. 57 described above.

The configuration example of the packet content in FIG. 58 illustrates aconfiguration example of packet contents in the case of “d_picture=1”,that the mode is for double picture, “picture_reorder=0”, and the Npicture is temporally first and the N+1 picture is later. Note that thisexample indicates a case where “partition_count=1111” and the partitionregion is “16”.

In this case, second disparity information for the entire picturedisplay screen of the N picture, i.e., “Max_disparity_in_picture” whichis the maximum disparity information (Disparity value) of the entirepicture, is disposed at the second byte (PB2). Also, the first disparityinformation of each partition region of the N picture, i.e.,“Min_disparity_in_partition” which is the minimum disparity information(Disparity value) of each partition region, is disposed from the thirdbyte (PB3) to the eighteenth byte (PB18).

Also, in this case, second disparity information of the entire picturedisplay screen of the N+1 picture, i.e., the“Differential_max_disparity_in_picture” which is differential data ofthe maximum disparity information (disparity value) of the entirepicture, is disposed from the third bit to the 0'th bit of thenineteenth byte (PB19). First disparity information of each partitionregion in the N+1 picture, i.e., the“Differential_min_disparity_in_partition” which is the differentialvalue of the minimum disparity information (disparity value) of eachpartition region, is disposed from the 20th byte (PB20) to the 27'thbyte (PB27).

Note that while a diagram and detailed description will be omitted, aconfiguration example of packet contents of “d_picture=1”, doublepicture mode, “picture_reorder=1”, and the case where N+1 picture istemporally first and N picture is later, corresponds to theconfiguration example in FIG. 52 described above.

Note that with this newly defined 3DDS packet, the bit width of “Payloadlength” in the packet header can take more than 5 bits, and the size ofthe packet contents that continue from this packet header can beincreased.

FIG. 59 illustrates a configuration example of packet contents in thecase thereof. The uppermost 5 bits of “3D_Metadata_length” are disposedfrom the fourth bit to the 0'th bit of the 0'th byte (PB0). Also, thelowest 8 bits of “3D_Metadata_length” are disposed from the seventh bitto the 0'th bit of the first byte (PB0).

Also, “partition_type” is disposed from the second bit to the 0'th bitof the second byte (PB2). Also, “partition_count” is disposed from theseventh bit to the 0'th bit of the third byte. Also, the disparityinformation (first disparity information, second disparity information)is disposed at the fourth byte (PB4) and thereafter.

“(3) Example of Using Active Space Region”

A method to transmit disparity information in each partition region foreach picture of the image data, using an active space region will bedescribed.

FIG. 60 illustrates a 3D video format of a frame packing method, whichis one TMDS transmitted data configuration for stereoscopic image data.This 3D video format is a format for transmitting image data for theleft eye (L) and right eye (R), as stereoscopic image data, with aprogressive format.

With this 3D video format, the transmission of image data in 1920×1080pand 1080×720p pixel formats are performed, as image data of the left eye(L) and right eye (R). Note that FIG. 60 illustrates an example wherethe left eye (L) image data and the right eye (R) image data are each1920 lines 1080 pixels.

With this 3D video format, transmitted data is generated to have unitsof video field segments that are segmented by the verticalsynchronization signal, including a horizontal blanking period (Hblank),vertical blanking period (Vblank), and active video space(Hactive×Vactive). In this 3D video format, the active video space hastwo active video regions (Active video) and in between thereof has oneactive space region. Left eye (L) image data is disposed in the firstactive video region, and the right eye (R) image data is disposed in thesecond active video region. The active video space has two active videoregions (Active video) as primary picture regions, and in betweenthereof has one active space region as an auxiliary picture region.

FIG. 61 illustrates an example of a packet configuration for HDMI VendorSpecific InfoFrame in the case of using an active space region. “ActiveSpace Enable” is defined in the second bit of the fifth byte (PB5) whichis a reserve (Reservedbit) in the current state, and this 1-bitinformation is set as “1”. Additionally, disparity information in eachpartition region of each picture of the image data is inserted into anactive space region, which in the current state is set as reserve(Reserved).

In this case, inserting a portion of the packet contents without changeof the newly defined 3DDS packet described above, for example, in theactive space region, may be conceived (see FIG. 57, FIG. 58). However,inserting in other forms can also be performed.

An active space region makes up an active video region in which left eyeimage data and right eye image data is disposed, as well as an activevideo space (see FIG. 60). Now, the active video region makes up theprimary picture region, and the active space region makes up theauxiliary picture region. The active space region varies by the imagesize of the video, but in the case of an image size of 1920*1080, holdscapacity of 45 lines (86400 bytes) per frame.

Note that in the description above, an example is illustrated where theset-top box 200 can select a mode for signal picture or double picture.However, instead of a double picture mode, a multiple picture mode maybe conceived, where the number of pictures can be optionally selected.Also, a case where the number of mode that can be selected is three ormore may be conceived. In that case, the number of partition regions maybe modified to an appropriate number on the HDMI source so thattransmissions can be made within the provided bands.

As described above, with the image transmitting/receiving system 10illustrated in FIG. 1, the broadcast station 100 inserts the firstdisparity information and second disparity information obtained for eachpredetermined picture of the image data into the video stream, andtransmits. Now, the first disparity information is disparity informationcorresponding to the nearest object playing position in a predeterminedregion of the picture display screen, and the second disparityinformation is disparity information corresponding to the farthestobject playing position in a predetermined region of the picture displayscreen. Therefore, with the set-top box 200 on the receiving side, forexample a check is performed as to whether or not the disparity anglesare within a predetermined range that do not harm the health of theviewer, based on first disparity information and second disparityinformation, and the left eye image data and right eye image data can bereconfigured as needed.

Also, with the image transmitting/receiving system 10 illustrated inFIG. 1, in the event that disparity information of each partition regionis transmitted from the set-top box 200 to the television receivingdevice 300, a newly defined 3DDS packet, for example, is used. In thiscase, the disparity information is inserted into the content portion ofthe 3DDS packet, which identification information to identify thedisparity information is inserted into the header portion of this 3DDSpacket, and this is transmitted to the television receiving device 300.Therefore, disparity information can be efficiently transmitted to thetelevision receiving device 300.

2. Modifications

Note that according to the above-described embodiment, an imagetransmitting/receiving system 10 that is made up of a broadcast station100, set-top box 200, and television receiving device 300, has beenillustrated. However, as illustrated in FIG. 62, an imagetransmitting/receiving system 10A that is made up of a broadcast station100 and television receiving device 300A may also be conceived.

FIG. 63 illustrates a configuration example of a television receivingdevice 300A. In FIG. 63 herein, portions that correspond to FIG. 28 havethe same reference numerals appended thereto, and detailed descriptionthereof will be omitted. A television (TV) graphics generating unit 219Ais similar to the set-top box (STB) graphics generating unit 219 of theset-top box 200 in FIG. 28, and generates an OSD or application, orgraphics data such as EPG.

From the overlaying unit 217, left eye image data onto which subtitlesand graphics for left eye display are overlaid is obtained, and righteye image data onto which subtitles and graphics for right eye displayare overlaid is obtained. This image data is send to a processing unitfor stereoscopic image display, and stereoscopic image display isperformed. Also, with the channel mixing unit 244, audio data for eachchannel to realize a 5.1 ch sound or the like, for example, is generatedas to the audio data. This audio data is supplied to a speaker, forexample, and audio output that matches the stereoscopic image display isperformed.

Detailed description will be omitted, but other than the televisionreceiving device 300A illustrated in FIG. 63 is configured similar tothe set-top box 200 in FIG. 28, and operates similarly.

Also, according to the above-described embodiment, an example isillustrated where disparity information in the horizontal direction foreach partition region of a picture display screen is transmitted asimage data related information with the newly defined 3DDS packet (seeFIG. 56 through FIG. 58). However, by using the 3DDS packet, other imagedata related information may be transmitted also.

For example, in addition to the conventional minimum_disparity values inthe horizontal direction, by sending minimum_disparity values in thevertical direction, correcting control of 3D disparity can be performedon the television receiving device side. This function can be performedby transmitting horizontal_minimum_disparity in picture increments andvetical_minimum_disparity in picture increments with a separate 3DDSpacket.

Also, for example, in the case that the value of the disparityinformation is automatically updated, when graphics of a certain size isdisplayed near in the depth direction, the size thereof is near butfeels small. On the other hand, if graphics of the same size isdisplayed far in the depth direction, the size thereof is far but feelslarge.

In order to inhibit such depth paradox, the scale is taken of the depthdynamic range, and the graphics are scaled and then overlaid onto theimage on the television receiving device side which overlays thegraphics accordingly. This function can be performed by transmittingvalues of the depth dynamic range between the minimum_disparity andmaximum_disparity at each portion that has been localized within thepicture, which have been scaled linearly or non-linearly, with aseparate 3DDS packet.

Also, according to the above-described embodiment, the set-top box 200and television receiving device 300 are illustrated as being connectedby an HDMI digital interface. However, it goes without saying that evenin a case where these are connected by a digital interface (includeswireless as well as cabled) similar to the HDMI digital interface, thepresent technology can be similarly applied.

Also, according to the above-described embodiment, as a method totransmit disparity information from the set-top box 200 to thetelevision receiving device 300, a method of using HDMI Vendor SpecificInfoFrame is described. Besides this, a method to use active space, andfurther, transmitting through a bi-directional communication path madeup of an HPD line 86 (HEAC-line) and utility line 88 (HEAC+ line) may beconceived.

Also, according to the above-described embodiment, an example isillustrated of transmitting disparity information from the set-top box200 to the television receiving device 300 with an HDMI interface.However, it goes without saying that, technology to transmit disparityinformation through an HDMI interface in this way can also be applied tocombinations of other source devices and sink devices. For example, as asource device, a disc player such as BD or DVD or the like, and further,a gaming device or the like may also be conceived, and as a sink device,a monitor device, projector device, or the like may also be conceived.

Also, according to the above-described embodiment, an example isillustrated where a container is a transport stream (MPEG-2 TS).However, the present technology can be similarly applied to a systemhaving a configuration where a network such as the Internet or the likeis used, and distributed to a receiving terminal. Of Internetdistribution, MP4 and containers of other formats are often used fordistribution.

That is to say, as a container, containers of various types of formats,such as a transport stream (MPEG-2 TS) used in digital broadcastingstandards, MP4 that is used in Internet distribution, and the like maybe used. Also, an application where one method for supplying servicecontent is divided into a plurality, and each is performed with separatetransmitted formats, which is to say, a case where one of the views istransmission by electronic waves and the other view is transmission bythe Internet is also applicable.

Also, the present technology can take configuration such as describedbelow.

(1) A transmitting apparatus including

-   -   a data packet generating unit to generate a data packet made up        of a header portion and a content portion; and    -   a transmitting unit to correlate the data packet to image data        and transmit to an external device,    -   wherein the data packet generating unit inserts related        information of the image data into the content portion, and        inserts identification information to identify the type of the        related information into the header portion.

(2) The transmitting apparatus according to (1) above, wherein the datapacket generating unit determines the size of the content portionaccording to the data amount of the related information that is insertedinto the content portion, and inserts size information indicating thedetermined size into the header portion.

(3) The transmitting apparatus according to (1) or (2) above, whereinthe data packet generating unit generates the data packet for each of apredetermined number of pictures of the image data.

(4) The transmitting apparatus according to (3) above,

-   -   wherein the image data is left eye image data and right eye        image data which configures a stereoscopic image; and    -   wherein the related information is other disparity information        as to one of a left eye image and right eye image, and is        representative disparity information for each predetermined        region of a picture display screen.

(5) The transmitting apparatus according to (4) above, wherein a firstdisparity information corresponding to the nearest object playingposition in a predetermined region is included in the representativedisparity information of each predetermined region.

(6) The transmitting apparatus according to (4) above, wherein the firstdisparity information corresponding to the nearest object playingposition in a predetermined region, and a second disparity informationcorresponding to the farthest object playing position in a predeterminedregion, is included in the representative disparity information of eachpredetermined region.

(7) The transmitting apparatus according to one of (4) through (6)above, wherein the data packet generating unit inserts therepresentative disparity information into the content portion asabsolute value data.

(8) The transmitting apparatus according to one of (1) through (7)above, wherein the transmitting unit inserts the data packet into ablanking period of the image data, and transmits to the external device.

(9) The transmitting apparatus according to Claim 1 in one of (1)through (7) above,

-   -   wherein the transmitting unit generates transmission data in        increments of video field segments that include horizontal        blanking periods and vertical blanking periods segmented by the        vertical synchronizing signal, and active video spaces having        primary picture regions and auxiliary picture regions, and        transmits to an external device, and    -   wherein image data is distributed to the primary picture        regions, and the data packets are distributed to the auxiliary        picture regions.

(10) A transmitting method including

-   -   a data packet generating step to generate a data packet made up        of a header portion and a content portion; and    -   a transmitting step to correlate the data packet to image data        and transmit to an external device,    -   wherein in the data packet generating step, related information        of the image data is inserted into the content portion, and        identification information to identify the type of the related        information is inserted into the header portion.

(11)

A transmitting apparatus including

-   -   an image data obtaining unit to obtain left eye image data and        right eye image data that configures a stereoscopic image;    -   a disparity information obtaining unit to obtain representative        disparity information which is the other disparity information        as to one of the left eye image and right eye image, for each        predetermined picture of the image data, and which is in each        partition region corresponding to a partition pattern of a        picture display screen;    -   a disparity information inserting unit to insert the        representative disparity information for each partition region        into the video stream obtained by the image data having been        encoded; and    -   an image data transmitting unit to transmit a container of a        predetermined format that includes a video stream in which the        disparity information has been inserted.

(12) The transmitting apparatus according to (11) above, furtherincluding

-   -   a pattern selecting unit to select predetermined partition        patterns from multiple partition patterns,    -   wherein the disparity information obtaining unit obtains        representative disparity information in each partition region        corresponding to the selected predetermined partition pattern of        the picture display screen.

(13) The transmitting apparatus according to (11) above, wherein thefirst disparity information corresponding to the nearest object playingposition in a partition region is included in the representativedisparity information in each of the partition regions.

(14) The transmitting apparatus according to (11) above, wherein thefirst disparity information corresponding to the nearest object playingposition in a partition region, and the second disparity informationcorresponding to the farthest object playing position in a partitionregion, is included in the representative disparity information in eachof the partition regions.

(15) The transmitting apparatus according to one of (11) through (14)above, wherein the disparity information inserting unit inserts therepresentative disparity information into the video stream, as absolutevalue data.

(16) A transmitting method, including

-   -   an image data obtaining step to obtain left eye image data and        right eye image data that configure a stereoscopic image;    -   a disparity information obtaining step to obtain representative        disparity information which is the other disparity information        as to one of the left eye image and right eye image for each        predetermined picture of the image data, and which is in each        partition region corresponding to a partition pattern of a        picture display screen;    -   a disparity information inserting step to insert the        representative disparity information in each partition region        into a video stream which is obtained by the image data having        been encoded; and    -   an image data transmitting step to transmit a container of a        predetermined format that includes the video stream into which        the disparity information has been inserted.

(17) A receiving device including

-   -   an image data receiving unit to receive a container of a        predetermined format that includes a video stream,    -   wherein the video stream is obtained by left eye image data and        right eye image data that configure a stereoscopic image having        been encoded; and    -   wherein representative disparity information which is the other        disparity information as to one of the left eye image and right        eye image, at each partition region corresponding to a partition        pattern of a picture display screen, is inserted into the video        stream for each picture of the image data;    -   the receiving apparatus further including    -   an information obtaining unit that obtains the left eye image        data and right eye image data from the video stream included in        the container, while obtaining representative information for        each partition region of each picture of the image data;    -   an information smoothing unit that performs smoothing processing        in the temporal axis direction as to the representative        disparity information for each partition region of each of the        picture;    -   a graphics data generating unit that generates graphics data to        display graphics on an image; and    -   an image data processing unit that uses the obtained image data        and the smoothed disparity information and the generated        graphics data, appends disparity corresponding to the display        position of the graphics for each picture to the graphics that        overlay a left eye image and right eye image, and obtains left        eye image data onto which the graphics have been overlaid and        right eye image data onto which the graphics have been overlaid.

Primary features of the present technology are related information ofimage data, for example disparity information or the like, beinginserted into a content portion of the data packet, while identificationinformation to identify the type of related information is inserted intothe header portion of this data packet, and this being transmitted to anexternal device, whereby related information of image data such asdisparity information or the like can be efficiently transmitted (seeFIG. 56 through FIG. 57). Also, by transmitting disparity information asabsolute value data, positive and negative sign bits are not needed, andthe dynamic range of the disparity information can be expanded (see FIG.15 through FIG. 17).

REFERENCE SIGNS LIST

-   -   10, 10A image transmitting/receiving system    -   100 broadcast station    -   110, 110A transmission data generating unit    -   111L, 111R image data output unit    -   112L, 112 scaler    -   113 video encoder    -   114 multiplexer    -   115 disparity data generating unit    -   116 subtitle data output unit    -   117 subtitle encoder    -   118 audio data output unit    -   119 audio encoder    -   120 partition pattern selecting unit    -   200 set-top box    -   211 container buffer    -   212 demultiplexer    -   213 coded buffer    -   214 video decoder    -   215 decoded buffer    -   216 scaler    -   217 overlaying unit    -   218 disparity information buffer    -   219 set-top box (STB) graphics buffer    -   219A television (TV) graphics buffer    -   220 depth control unit    -   221 graphics buffer    -   231 coded buffer    -   232 subtitle decoder    -   233 pixel buffer    -   234 subtitle disparity information buffer    -   235 subtitle display control unit    -   241 coded buffer    -   242 audio decoder    -   243 audio buffer    -   244 channel mixing unit    -   251 HDMI transmitting unit    -   262 disparity information control unit    -   263 L/R reconfiguring unit    -   271 maximum value obtaining unit    -   272 minimum value obtaining unit    -   273 disparity angle checking unit    -   274 disparity information correcting unit    -   281 frame memory    -   282 subtractor    -   283 order determining unit    -   300, 300A television receiving device    -   311 HDMI receiving unit    -   312 scaler    -   313 overlaying unit    -   314 depth control unit    -   315 graphics buffer    -   316 television (TV) graphics generating unit    -   317 audio processing unit    -   400 HDMI cable

The invention claimed is:
 1. A transmitting apparatus comprising:circuitry configured to: generate a data packet made up of a headerportion and a content portion; correlate the data packet to image dataand transmit to an external device; and insert related information ofthe image data into the content portion, and insert identificationinformation to identify a type of the related information into theheader portion, wherein the related information comprises firstdisparity information and second disparity information, and wherein thefirst disparity information corresponds to a nearest object playingposition in each partition region obtained by partitioning a picturedisplay screen into a plurality of partition regions and the seconddisparity information corresponds to a farthest object playing positionover an entire picture display screen.
 2. The transmitting apparatusaccording to claim 1, wherein the circuitry is configured to determinethe size of the content portion according to a data amount of therelated information that is inserted into the content portion, andinsert size information indicating the determined size into the headerportion.
 3. The transmitting apparatus according to claim 1, wherein thecircuitry is configured to generate the data packet for each of apredetermined number of pictures of the image data.
 4. The transmittingapparatus according to claim 3, wherein the image data is left eye imagedata and right eye image data which configures a stereoscopic image, andwherein the related information is other disparity information as to oneof a left eye image and a right eye image, and is representativedisparity information for each predetermined region of the picturedisplay screen.
 5. The transmitting apparatus according to claim 4,wherein the first disparity information corresponding to the nearestobject playing position in a predetermined region is included in therepresentative disparity information of each predetermined region. 6.The transmitting apparatus according to claim 4, wherein the firstdisparity information and the second disparity information are includedin the representative disparity information of each predeterminedregion.
 7. The transmitting apparatus according to claim 4, wherein thecircuitry is configured to insert the representative disparityinformation into the content portion as absolute value data.
 8. Thetransmitting apparatus according to claim 1, wherein the circuitry isconfigured to insert the data packet into a blanking period of the imagedata, and transmit to the external device.
 9. The transmitting apparatusaccording to claim 1, wherein the circuitry is configured to generatetransmission data in increments of video field segments that includehorizontal blanking periods and vertical blanking periods segmented bythe vertical synchronizing signal, and active video spaces havingprimary picture regions and auxiliary picture regions, and transmit toan external device, and wherein the image data is distributed to theprimary picture regions, and the data packets are distributed to theauxiliary picture regions.
 10. A transmitting method comprising: a datapacket generating step to generate a data packet made up of a headerportion and a content portion; and a transmitting step to correlate thedata packet to image data and transmit to an external device, wherein inthe data packet generating step, related information of the image datais inserted into the content portion, and identification information toidentify a type of the related information is inserted into the headerportion, wherein the related information comprises first disparityinformation and second disparity information, and wherein the firstdisparity information corresponding to a nearest object playing positionin each partition region obtained by partitioning a picture displayscreen into a plurality of partition regions and the second disparityinformation corresponding to a farthest object playing position over anentire picture display screen.
 11. A transmitting apparatus comprising:circuitry configured to: obtain left eye image data and right eye imagedata that configures a stereoscopic image; obtain representativedisparity information which is the other disparity information as to oneof a left eye image and a right eye image, for each predeterminedpicture of image data, and which is in each partition regioncorresponding to a partition pattern of a picture display screen,wherein the representative disparity information comprises firstdisparity information and second disparity information, wherein thefirst disparity information corresponds to a nearest object playingposition in each partition region obtained by partitioning a picturedisplay screen into a plurality of partition regions and the seconddisparity information corresponds to a farthest object playing positionover an entire picture display screen; insert the representativedisparity information for each partition region into a video streamobtained by the image data having been encoded; and transmit a containerof a predetermined format that includes the video stream in which thedisparity information has been inserted.
 12. The transmitting apparatusaccording to claim 11, wherein the circuitry is configured to: selectpredetermined partition patterns from a plurality of partition patterns,obtain the representative disparity information in each partition regioncorresponding to the selected predetermined partition pattern of thepicture display screen.
 13. The transmitting apparatus according toclaim 11, wherein the representative disparity information is insertedinto the video stream as absolute value data.
 14. A transmitting method,comprising: an image data obtaining step to obtain left eye image dataand right eye image data that configure a stereoscopic image; adisparity information obtaining step to obtain representative disparityinformation which is the other disparity information as to one of a lefteye image and a right eye image for each predetermined picture of theimage data, and which is in each partition region corresponding to apartition pattern of a picture display screen, wherein therepresentative disparity information comprises first disparityinformation and second disparity information, wherein the firstdisparity information corresponding to a nearest object playing positionin each partition region obtained by partitioning a picture displayscreen into a plurality of partition regions and the second disparityinformation corresponding to a farthest object playing position over anentire picture display screen; a disparity information inserting step toinsert the representative disparity information in each partition regioninto a video stream which is obtained by the image data having beenencoded; and an image data transmitting step to transmit a container ofa predetermined format that includes the video stream into which thedisparity information has been inserted.